metadata

tags:
  - autotrain
  - text-generation-inference
  - text-generation
  - peft
library_name: transformers
widget:
  - messages:
      - role: user
        content: What is your favorite condiment?
license: other

Model Trained Using AutoTrain

This model was trained using AutoTrain. For more information, please visit AutoTrain.

Usage

pip install peft

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import AutoPeftModelForCausalLM, PeftConfig

model_id = "Aryan-401/phi-3-mini-4k-instruct-finetune-guanaco"
peft_model=AutoPeftModelForCausalLM.from_pretrained(model_id)

model = peft_model.merge_and_unload()
tokenizer = AutoTokenizer.from_pretrained(model_id)

messages = [
    {"role": "user", "content": "What is the Value of Pi?"}
]
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = model.to(device).eval()

input_ids = tokenizer.apply_chat_template(conversation=messages, tokenize=True, add_generation_prompt=True, return_tensors='pt')
output_ids = model.generate(input_ids.to(device), max_length= 1000)
response = tokenizer.decode(output_ids[0][input_ids.shape[1]:], skip_special_tokens=True)

print(response)