jordimas commited on
Commit
1b1d426
1 Parent(s): d5d419f

Initial version

Browse files
Files changed (5) hide show
  1. .gitattributes +2 -0
  2. README.md +58 -3
  3. model.bin +3 -0
  4. shared_vocabulary.txt +0 -0
  5. sp_m.model +3 -0
.gitattributes CHANGED
@@ -25,3 +25,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
25
  *.zip filter=lfs diff=lfs merge=lfs -text
26
  *.zstandard filter=lfs diff=lfs merge=lfs -text
27
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
25
  *.zip filter=lfs diff=lfs merge=lfs -text
26
  *.zstandard filter=lfs diff=lfs merge=lfs -text
27
  *tfevents* filter=lfs diff=lfs merge=lfs -text
28
+ model.bin filter=lfs diff=lfs merge=lfs -text
29
+ sp_m.model filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,3 +1,58 @@
1
- ---
2
- license: mit
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - ca
4
+ - de
5
+
6
+ tags:
7
+ - translation
8
+
9
+ library_name: opennmt
10
+ license: mit
11
+ metrics:
12
+ - bleu
13
+
14
+ inference: false
15
+ ---
16
+
17
+ ### Introduction
18
+
19
+ Catalan - German translation model for OpenNMT. These are the same models that we have in production at https://www.softcatala.org/traductor/.
20
+ The models are quantified for low latency.
21
+
22
+ ### Usage
23
+
24
+ Install the necessary dependencies:
25
+
26
+
27
+ ```bash
28
+ pip3 install ctranslate2 pyonmttok
29
+ ```
30
+
31
+
32
+ Simple tokenization & translation using Python:
33
+
34
+
35
+ ```python
36
+ import ctranslate2
37
+ import pyonmttok
38
+ from huggingface_hub import snapshot_download
39
+ model_dir = snapshot_download(repo_id="softcatala/opennmt-cat-deu", revision="main")
40
+
41
+ tokenizer=pyonmttok.Tokenizer(mode="none", sp_model_path = model_dir + "/sp_m.model")
42
+ tokenized=tokenizer.tokenize("Hola món")
43
+
44
+ translator = ctranslate2.Translator(model_dir)
45
+ translated = translator.translate_batch([tokenized[0]])
46
+ print(tokenizer.detokenize(translated[0][0]['tokens']))
47
+ ```
48
+
49
+ ## Benchmarks
50
+
51
+ | testset | BLEU |
52
+ |---------------------------------------|-------|
53
+ | test dataset (from train/dev/test) | 30.6 |
54
+ | Flores101 dataset | 21.6 |
55
+
56
+ ## Additional information
57
+ * https://github.com/Softcatala/nmt-models
58
+ * https://github.com/Softcatala/parallel-catalan-corpus
model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ed303d86a25183a69224f31d55d903c8b7b7fb4f1bcfbf754ad6b75d8dd238b1
3
+ size 122582230
shared_vocabulary.txt ADDED
The diff for this file is too large to render. See raw diff
 
sp_m.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0ada14f322a2553be3bd3c729632bd7c45bda8f564ba5d714719b0ff42fbeb97
3
+ size 1193792