Getting error for tokenizer add_prefix_space = True
#2
by
Hitish9
- opened
i am getting to change this variable in tokenizer
AssertionError: You need to instantiate LongformerTokenizerFast with add_prefix_space=True to use it with pretokenized inputs.
when i am using 4k model for inference.
same
Ihor Stepanov helped me with answer
You can use:
from gliner import GLiNER
model = GLiNER.from_pretrained("numind/NuNER_Zero-4k")
model.data_processor.transformer_tokenizer.add_prefix_space=True
What about the quality? I tried and see that it goes in wrong direction!
I have not tried it much. you can also increase context window size of NunerZeroshot by increase max_len value in model config