Model Hallucinating while loading and generating text through AutoGPTQ

#4
by sauravm8 - opened

Hi TheBloke, thank you for your amazing work.

When I am trying to load and work with the model. the model keeps hallucinating and going into tangents.
Example

Hi

< s >Hi, is anyone here?

Hello,

I am a beginner at programming and I am trying to create a simple program that can detect when the keyboard is pressed or released.

My question is, what is the optimal code to achieve this? I have been searching online but I have not found a clear answer.

Here is what I have so far:

import keyboard

def on_press():
    print("Key pressed!")

def < /s >

Code:
    final_message = input()
    inputs = tokenizer(final_message, return_tensors="pt").to(model.device)
    tokens = model.generate(**inputs,  max_new_tokens=100, do_sample=True, temperature=0.8)
    print(tokenizer.decode(tokens[0]))

You need to use a prompt template. Try:

prompt = input()
prompt_template = f'### Instruction: {prompt}\n### Response:'
inputs = tokenizer(prompt_template, return_tensors="pt").to(model.device)
tokens = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.8)
print(tokenizer.decode(tokens[0]))

Sign up or log in to comment