THUDM

/

chatglm2-6b

like 2.03k

Follow

Knowledge Engineering Group (KEG) & Data Mining at Tsinghua University 1,189

使用最新版Transformers对Tokenizer进行加载的时候会报错

AttributeError: 'ChatGLMTokenizer' object has no attribute 'sp_tokenizer'. Did you mean: '_tokenize'?
    return self.sp_tokenizer.num_tokens

解决办法：使用修复版的Tokenizer

将THUDM/chatglm-6b替换为PiaoYang/chatglm-6b即可。同时如果有revision参数，需要去掉。

tokenizer = AutoTokenizer.from_pretrained("PiaoYang/chatglm-6b", trust_remote_code=True)