alistairewj commited on
Commit
9048f31
1 Parent(s): e1354b7

add model_max_length

Browse files

without specifying the model_max_length for the tokenizer defaults to a very large int, and inference crashes

I believe this may be due to a recent change to transformers - bert-base-uncased config was updated 2 months ago: https://huggingface.co/google-bert/bert-base-uncased/commit/86b5e0934494bd15c9632b12f734a8a67f723594

Files changed (1) hide show
  1. tokenizer_config.json +1 -3
tokenizer_config.json CHANGED
@@ -1,3 +1 @@
1
- {
2
- "do_lower_case": true
3
- }
 
1
+ {"do_lower_case": true, "model_max_length": 512}