update tokenizer for compatibility with new transformers bbb5eac verified katuni4ka commited on 7 days ago
fix can't set attribute 'eos_token' when loading the saved tokenizer (#27) 6d10497 zRzRzRzRzRzRzR hiyouga commited on Dec 29, 2023