KoichiYasuoka
commited on
Commit
•
8b160e2
1
Parent(s):
b6984e9
re-initialize
Browse files
README.md
ADDED
@@ -0,0 +1,87 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
language:
|
3 |
+
- "lzh"
|
4 |
+
tags:
|
5 |
+
- "classical chinese"
|
6 |
+
- "literary chinese"
|
7 |
+
- "ancient chinese"
|
8 |
+
- "token-classification"
|
9 |
+
- "pos"
|
10 |
+
license: "mit"
|
11 |
+
pipeline_tag: "token-classification"
|
12 |
+
widget:
|
13 |
+
- text: "不入虎穴不得虎子"
|
14 |
+
---
|
15 |
+
|
16 |
+
[![Current PyPI packages](https://badge.fury.io/py/suparkanbun.svg)](https://pypi.org/project/suparkanbun/)
|
17 |
+
|
18 |
+
# SuPar-Kanbun
|
19 |
+
|
20 |
+
Tokenizer, POS-Tagger and Dependency-Parser for Classical Chinese Texts (漢文/文言文) with [spaCy](https://spacy.io), [Transformers](https://huggingface.co/transformers/) and [SuPar](https://github.com/yzhangcs/parser).
|
21 |
+
|
22 |
+
## Basic usage
|
23 |
+
|
24 |
+
```py
|
25 |
+
>>> import suparkanbun
|
26 |
+
>>> nlp=suparkanbun.load()
|
27 |
+
>>> doc=nlp("不入虎穴不得虎子")
|
28 |
+
>>> print(type(doc))
|
29 |
+
<class 'spacy.tokens.doc.Doc'>
|
30 |
+
>>> print(suparkanbun.to_conllu(doc))
|
31 |
+
# text = 不入虎穴不得虎子
|
32 |
+
1 不 不 ADV v,副詞,否定,無界 Polarity=Neg 2 advmod _ Gloss=not|SpaceAfter=No
|
33 |
+
2 入 入 VERB v,動詞,行為,移動 _ 0 root _ Gloss=enter|SpaceAfter=No
|
34 |
+
3 虎 虎 NOUN n,名詞,主体,動物 _ 4 nmod _ Gloss=tiger|SpaceAfter=No
|
35 |
+
4 穴 穴 NOUN n,名詞,固定物,地形 Case=Loc 2 obj _ Gloss=cave|SpaceAfter=No
|
36 |
+
5 不 不 ADV v,副詞,否定,無界 Polarity=Neg 6 advmod _ Gloss=not|SpaceAfter=No
|
37 |
+
6 得 得 VERB v,動詞,行為,得失 _ 2 parataxis _ Gloss=get|SpaceAfter=No
|
38 |
+
7 虎 虎 NOUN n,名詞,主体,動物 _ 8 nmod _ Gloss=tiger|SpaceAfter=No
|
39 |
+
8 子 子 NOUN n,名詞,人,関係 _ 6 obj _ Gloss=child|SpaceAfter=No
|
40 |
+
|
41 |
+
>>> import deplacy
|
42 |
+
>>> deplacy.render(doc)
|
43 |
+
不 ADV <════╗ advmod
|
44 |
+
入 VERB ═══╗═╝═╗ ROOT
|
45 |
+
虎 NOUN <╗ ║ ║ nmod
|
46 |
+
穴 NOUN ═╝<╝ ║ obj
|
47 |
+
不 ADV <════╗ ║ advmod
|
48 |
+
得 VERB ═══╗═╝<╝ parataxis
|
49 |
+
虎 NOUN <╗ ║ nmod
|
50 |
+
子 NOUN ═╝<╝ obj
|
51 |
+
```
|
52 |
+
|
53 |
+
`suparkanbun.load()` has two options `suparkanbun.load(BERT="roberta-classical-chinese-base-char",Danku=False)`. With the option `Danku=True` the pipeline tries to segment sentences automatically. Available `BERT` options are:
|
54 |
+
|
55 |
+
* `BERT="roberta-classical-chinese-base-char"` utilizes [roberta-classical-chinese-base-char](https://huggingface.co/KoichiYasuoka/roberta-classical-chinese-base-char) (default)
|
56 |
+
* `BERT="roberta-classical-chinese-large-char"` utilizes [roberta-classical-chinese-large-char](https://huggingface.co/KoichiYasuoka/roberta-classical-chinese-large-char)
|
57 |
+
* `BERT="guwenbert-base"` utilizes [GuwenBERT-base](https://huggingface.co/ethanyt/guwenbert-base)
|
58 |
+
* `BERT="guwenbert-large"` utilizes [GuwenBERT-large](https://huggingface.co/ethanyt/guwenbert-large)
|
59 |
+
* `BERT="sikubert"` utilizes [SikuBERT](https://huggingface.co/SIKU-BERT/sikubert)
|
60 |
+
* `BERT="sikuroberta"` utilizes [SikuRoBERTa](https://huggingface.co/SIKU-BERT/sikuroberta)
|
61 |
+
|
62 |
+
## Installation for Linux
|
63 |
+
|
64 |
+
```sh
|
65 |
+
pip3 install suparkanbun --user
|
66 |
+
```
|
67 |
+
|
68 |
+
## Installation for Cygwin64
|
69 |
+
|
70 |
+
Make sure to get `python37-devel` `python37-pip` `python37-cython` `python37-numpy` `python37-wheel` `gcc-g++` `mingw64-x86_64-gcc-g++` `git` `curl` `make` `cmake` packages, and then:
|
71 |
+
```sh
|
72 |
+
curl -L https://raw.githubusercontent.com/KoichiYasuoka/CygTorch/master/installer/supar.sh | sh
|
73 |
+
pip3.7 install suparkanbun --no-build-isolation
|
74 |
+
```
|
75 |
+
|
76 |
+
## Installation for Jupyter Notebook (Google Colaboratory)
|
77 |
+
|
78 |
+
```py
|
79 |
+
!pip install suparkanbun
|
80 |
+
```
|
81 |
+
|
82 |
+
Try [notebook](https://colab.research.google.com/github/KoichiYasuoka/SuPar-Kanbun/blob/main/suparkanbun.ipynb) for Google Colaboratory.
|
83 |
+
|
84 |
+
## Author
|
85 |
+
|
86 |
+
Koichi Yasuoka (安岡孝一)
|
87 |
+
|