xgen-7b-8k-base / tokenization_xgen.py

Commit History

_convert_id_to_token
aa0c006

tianxie-sf commited on

add pad token and default eos token (#6)
e9a901f

rooa tianxie-sf commited on

.
d2cc612

rooa commited on

.
b9f4adf

rooa commited on

.
04d6513

rooa commited on