nicce/story-mixtral-8x7b-lora

Model Card for nicce/sexbot

This is a lora adapter for mistralai/Mixtral-8x7b-Instruct-v0.1

Testing the Model

To quickly test the model, you can run it on a GPU with the transformers / peft library:

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("nicce/sexbot")
model = AutoPeftModelForCausalLM.from_pretrained("nicce/sexbot").to("cuda") # if you get a CUDA out of memory error, try load_in_8bit=True

messages = [
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "Hi, can you please explain machine learning to me?"}
]

encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
generated_ids = model.generate(input_ids=model_inputs, min_new_tokens=10, max_new_tokens=300, do_sample=True, temperature=0.9, top_p=0.8)
decoded = tokenizer.batch_decode(generated_ids)

print(decoded[0])