Text-to-text Generation Models (LLMs, Llama, GPT, ...)
Collection
5130 items
•
Updated
•
12
Frequently Asked Questions
from transformers import AutoTokenizer
import transformers
import torch
model = "PrunaAI/Mistral-7B-Instruct-v0.2-bnb-4bit"
messages = [{"role": "user", "content": "What is a large language model?"}]
tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
"text-generation",
model=model,
device_map="auto",
)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
The license of the smashed model follows the license of the original model. Please check the license of the original model mistralai/Mistral-7B-Instruct-v0.2 before using this model which provided the base model. The license of the pruna-engine
is here on Pypi.