tokenizer.add_eos_token and maxlength

#15

by Gregorioz - opened Apr 23

Discussion

Gregorioz

Apr 23

•

edited Apr 23

In the old version of demo code, tokenization processes were tokenizer.add_eos_token = True and tokenizer(...,maxlength=maxlength-1,...). Recent update removed tokenizer.add_eos_token = True, and modified value of argument maxlengthfrom maxlength-1 to maxlength. What are the differences of two tokenization methods? Is there any risk since divergent embeddings are observed?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment