Commit History
update tokenizer for compatibility with new transformers (#64)
3afaed6
verified
fix for apply_chat_template
37f2196
verified
fix can't set attribute 'eos_token' when loading the saved tokenizer (#27)
6d10497
fix piece id out of range (#20)
456aa87
Add encode_special_tokens option
e46a148
duzx16
commited on
Implement new interface
1676f07
duzx16
commited on
Add system prompt
591fa87
duzx16
commited on
Implement new chat method
85ba2d2
duzx16
commited on