'ChatGLMTokenizer' object has attribute 'sp_tokenizer'

#109
by LavanyaSuresh - opened

I tried to convert the model to openvino , By using below comments ,

Safetensors Models are still missing from Chatglm-6B https://huggingface.co/THUDM/chatglm-6b/tree/main ,& working model https://huggingface.co/THUDM/chatglm3-6b/tree/main Please have a compare,

Still facing sp_tokenizer issue and Https response issue..
error_1.png

error_2.png

Usage:
python 3.10.11
Ov latest toolkit

Could you please to resolve the issue ..

我使用THUDM/chatglm-6b的tokenizer也遇到了类似的情况:
image.png
Python:3.11.10
transformers:4.45.2

Knowledge Engineering Group (KEG) & Data Mining at Tsinghua University org

降级到4.40,只有GLM-4有继续维护到这个版本

看了之前的帖子,用了新版本:PiaoYang/chatglm-6b
tokenizer = AutoTokenizer.from_pretrained("PiaoYang/chatglm-6b",trust_remote_code=True)
可以使用。

Sign up or log in to comment