Spaces:

Dovakiins
/

qwerrwe

Build error

App Files Files Community

qwerrwe / tests /test_tokenizers.py

Commit History

Support for additional_special_tokens (#1221) [skip ci]

25e037f
unverified

DreamGenX

winglian commited on Jan 31

Feat: Warns to add to modules_to_save when adding tokens or switching special_tokens (#787)

1ffa386
unverified

Nanobit commited on Dec 22, 2023

simplify `load_tokenizer`

efb3b2c

tmm1 commited on Aug 13, 2023

optionally define whether to use_fast tokenizer

47d601f

winglian commited on Jun 25, 2023