ryan-minato
commited on
Commit
โข
84ee4a0
1
Parent(s):
c4ac097
Update README.md
Browse files
README.md
CHANGED
@@ -26,3 +26,21 @@ Models previously compressed using zstd have been decompressed, allowing for imm
|
|
26 |
|
27 |
## Usage
|
28 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
26 |
|
27 |
## Usage
|
28 |
|
29 |
+
```python
|
30 |
+
from huggingface_hub import hf_hub_download
|
31 |
+
import vibrato
|
32 |
+
|
33 |
+
# Load tokenizer from `.cache/hf`
|
34 |
+
model_path = hf_hub_download("aha-org/vibrato-models-bsdonly", "<<model_name>>/system.dic")
|
35 |
+
with open(model_path, "rb") as f:
|
36 |
+
tokenizer = vibrato.Vibrato(f.read())
|
37 |
+
|
38 |
+
text = """\
|
39 |
+
ใๅๅไบใ ใจ!ใใซใผใณใฏใฉใผใซใๅซใใ ใ
|
40 |
+
ใไธ็พไบๅไธๅนดใใใฆใใใใ ใใ?ใ
|
41 |
+
ใไฝๅบฆใๅพนๅบ็ใซๆค็ฎใใพใใใใณใณใใฅใผใฟใๅฟใใใ
|
42 |
+
ใใพใกใใใชใใใใ็ญใใงใใ็็ดใชใจใใใใฟใชใใใฎใปใใง็ฉถๆฅตใฎ็ๅใไฝใงใใใใใใฃใฆใใชใใฃใใจใใใซๅ้กใใใใฎใงใใ
|
43 |
+
"""
|
44 |
+
|
45 |
+
tokenizer.tokenize(text)
|
46 |
+
```
|