Update README.md (#966)
Browse files
README.md
CHANGED
@@ -981,7 +981,7 @@ wandb_log_model:
|
|
981 |
|
982 |
##### Special Tokens
|
983 |
|
984 |
-
It is important to have special tokens like delimiters, end-of-sequence, beginning-of-sequence in your tokenizer's
|
985 |
|
986 |
```yml
|
987 |
special_tokens:
|
|
|
981 |
|
982 |
##### Special Tokens
|
983 |
|
984 |
+
It is important to have special tokens like delimiters, end-of-sequence, beginning-of-sequence in your tokenizer's vocabulary. This will help you avoid tokenization issues and help your model train better. You can do this in axolotl like this:
|
985 |
|
986 |
```yml
|
987 |
special_tokens:
|