baichuan-inc

/

Baichuan2-13B-Chat

like 422

Follow

Baichuan Intelligent Technology 258

RT
我使用了如下代码转换,但并没有生成tokenizer.json.

tokenizer = AutoTokenizer.from_pretrained("baichuan-inc/Baichuan2-13B-Chat", use_fast=False, trust_remote_code=True)
tokenizer.save_pretrained("./tokenizer/")

output

('./tokenizer/tokenizer_config.json',
 './tokenizer/special_tokens_map.json',
 './tokenizer/tokenizer.model',
 './tokenizer/added_tokens.json')