metadata
tags:
- autotrain
- text-generation-inference
- text-generation
- peft
library_name: transformers
widget:
- messages:
- role: user
content: What is your favorite condiment?
license: other
Model Trained Using AutoTrain
This model was trained using AutoTrain. For more information, please visit AutoTrain.
Usage
pip install peft
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import AutoPeftModelForCausalLM, PeftConfig
model_id = "Aryan-401/phi-3-mini-4k-instruct-finetune-guanaco"
peft_model=AutoPeftModelForCausalLM.from_pretrained(model_id)
model = peft_model.merge_and_unload()
tokenizer = AutoTokenizer.from_pretrained(model_id)
messages = [
{"role": "user", "content": "What is the Value of Pi?"}
]
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device).eval()
input_ids = tokenizer.apply_chat_template(conversation=messages, tokenize=True, add_generation_prompt=True, return_tensors='pt')
output_ids = model.generate(input_ids.to(device), max_length= 1000)
response = tokenizer.decode(output_ids[0][input_ids.shape[1]:], skip_special_tokens=True)
print(response)