traintogpb
commited on
Commit
•
6a245f2
1
Parent(s):
0d2c0ea
chore: fix model card
Browse files
README.md
CHANGED
|
@@ -17,12 +17,10 @@ pipeline_tag: translation
|
|
| 17 |
|
| 18 |
### Training Dataset
|
| 19 |
- [traintogpb/aihub-flores-koen-integrated-sparta-30k](https://huggingface.co/datasets/traintogpb/aihub-flores-koen-integrated-sparta-30k)
|
| 20 |
-
|
| 21 |
- Can translate in Enlgish-Korean (bi-directional)
|
| 22 |
|
| 23 |
### Prompt
|
| 24 |
- Template:
|
| 25 |
-
|
| 26 |
```python
|
| 27 |
prompt = f"Translate this from {src_lang} to {tgt_lang}\n### {src_lang}: {src_text}\n### {tgt_lang}:"
|
| 28 |
|
|
@@ -33,7 +31,6 @@ pipeline_tag: translation
|
|
| 33 |
- Issue:
|
| 34 |
The tokenizer of the model tokenizes the prompt below in different way with the prompt above.
|
| 35 |
Make sure to use the prompt proposed above.
|
| 36 |
-
|
| 37 |
```python
|
| 38 |
prompt = f"""Translate this from {src_lang} to {tgt_lang}
|
| 39 |
### {src_lang}: {src_text}
|
|
@@ -41,7 +38,6 @@ pipeline_tag: translation
|
|
| 41 |
|
| 42 |
>>> # DO NOT USE this prompt
|
| 43 |
```
|
| 44 |
-
|
| 45 |
And mind that there is no "space (`_`)" at the end of the prompt.
|
| 46 |
|
| 47 |
### Training
|
|
@@ -52,7 +48,6 @@ pipeline_tag: translation
|
|
| 52 |
|
| 53 |
### Usage (IMPORTANT)
|
| 54 |
- Should remove the EOS token (`<|endoftext|>`, id=46332) at the end of the prompt.
|
| 55 |
-
|
| 56 |
```python
|
| 57 |
# MODEL
|
| 58 |
plm_name = 'beomi/open-llama-2-ko-7b'
|
|
|
|
| 17 |
|
| 18 |
### Training Dataset
|
| 19 |
- [traintogpb/aihub-flores-koen-integrated-sparta-30k](https://huggingface.co/datasets/traintogpb/aihub-flores-koen-integrated-sparta-30k)
|
|
|
|
| 20 |
- Can translate in Enlgish-Korean (bi-directional)
|
| 21 |
|
| 22 |
### Prompt
|
| 23 |
- Template:
|
|
|
|
| 24 |
```python
|
| 25 |
prompt = f"Translate this from {src_lang} to {tgt_lang}\n### {src_lang}: {src_text}\n### {tgt_lang}:"
|
| 26 |
|
|
|
|
| 31 |
- Issue:
|
| 32 |
The tokenizer of the model tokenizes the prompt below in different way with the prompt above.
|
| 33 |
Make sure to use the prompt proposed above.
|
|
|
|
| 34 |
```python
|
| 35 |
prompt = f"""Translate this from {src_lang} to {tgt_lang}
|
| 36 |
### {src_lang}: {src_text}
|
|
|
|
| 38 |
|
| 39 |
>>> # DO NOT USE this prompt
|
| 40 |
```
|
|
|
|
| 41 |
And mind that there is no "space (`_`)" at the end of the prompt.
|
| 42 |
|
| 43 |
### Training
|
|
|
|
| 48 |
|
| 49 |
### Usage (IMPORTANT)
|
| 50 |
- Should remove the EOS token (`<|endoftext|>`, id=46332) at the end of the prompt.
|
|
|
|
| 51 |
```python
|
| 52 |
# MODEL
|
| 53 |
plm_name = 'beomi/open-llama-2-ko-7b'
|