Inference Error
I tried to use your code snippet with transformers library version==4.31.0, but all the model generates is this character "\x0f".
I was wondering if I should be using another version of transformers.
Is there any fix to this problem?
Hi, thanks for the response.
I´m using this versions of libraries:
torch == 1.11.0 and 2.0.0
transformers == 4.31.0
bitsandbytes == 0.41.0 (latest)
and I am running this code with cpu instead of cuda, which shouldn´t affect the result (?)
Is there anything else I can try to make the completion work?
Thanks again!
same error here
Same
Thanks for your feedback. Working on fixing it
Same error here. Using load_in_8bit and with auto device map:model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, load_in_8bit = True, device_map='auto')
Response to all inputs is:ed 10c30c30c30c300c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30c30-slash.