Inference Code
import os
os.environ["CUDA_VISIBLE_DEVICES"]="0"
from dataclasses import dataclass, field
from typing import Optional
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
BitsAndBytesConfig,
HfArgumentParser,
AutoTokenizer,
TrainingArguments,
)
from trl import SFTTrainer
from peft import (
prepare_model_for_kbit_training,
LoraConfig,
get_peft_model,
PeftModel
)
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype="bfloat16",
bnb_4bit_use_double_quant=False,
)
device_map = {"": 0}
tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-40b")
model = AutoModelForCausalLM.from_pretrained(
"tiiuae/falcon-40b", quantization_config=bnb_config, device_map=device_map, trust_remote_code=True
)
model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=False)
model_id = "smangrul/falcon-40B-int4-peft-lora-sfttrainer"
model = PeftModel.from_pretrained(model, model_id)
text = '### Human: Write a tweet celebrating the Apache-2 release of Falcon models which are generative Large Language Models (LLMs) on which you have been finetuned. Previously, it was under a bit of a restrictive license. Make the tweet punchy, energetic, exciting and marketable.### Assitant:'
outputs = model.generate(input_ids=tokenizer(text, return_tensors="pt").input_ids,
max_new_tokens=256,
temperature=0.7,
top_p=0.9,
do_sample=True)
print(tokenizer.batch_decode(outputs))
Example completions:
### Human: Write a tweet celebrating the Apache-2 release of Falcon models which are generative Large Language Models (LLMs) on which you have been finetuned.
Previously, it was under a bit of a restrictive license.
Make the tweet punchy, energetic, exciting and marketable.
### Assitant: The Apache-2 release of Falcon models is a huge milestone for the Open Source community!
🎉 Previously, Falcon was only available under a restrictive license, but now anyone can use and contribute to it.
\n\nFalcon is a large language
### Human: Compose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions.
### Assitant: Here is a sample travel blog post about a recent trip to Hawaii:\n\n#
A Guide to Cultural Experiences and Must-See Attractions in Hawaii\n\nIf you\'re looking for a tropical getaway, Hawaii is the perfect destination.
With its stunning beaches, lush rainforests, and rich culture, there\'s something for everyone in this beautiful state.
In this blog post, we\'ll explore some of the best cultural experiences and must-see attractions in Hawaii.\n\n##
Experience the Aloha Spirit\n\nOne of the best things about Hawaii is the people.
The locals are known for their hospitality and kindness, and you\'ll often hear them say "aloha" as a greeting.
The aloha spirit is all about treating others with respect and kindness, and it\'s a big part of what makes Hawaii such a special place.\n\n
## Visit a Luau\n\nOne of the most popular cultural experiences in Hawaii is the luau. A luau is a traditional Hawaiian feast that includes food, music, and dance.
It\'s a great way to experience the culture and learn more about Hawaiian history and traditions.
There are many different luau options available, so you can find one that fits your interests and budget.\n\n## Explore the Natural W'
### Human: what are Asimov\'s laws of robotics?.
### Assitant: Asimov\'s Laws of Robotics are a set of three rules devised by science fiction writer Isaac Asimov.
They were first introduced in his 1942 short story "Runaround", and were later expanded upon in his 1950 novel "I, Robot".
The laws are as follows:\n\n
* A robot may not injure a human being or, through inaction, allow a human being to come to harm.\n
* A robot must obey the orders given it by human beings, except where such orders would conflict with the First Law.\n
* A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws.\n\n
Asimov later added a "zeroth law" to his laws, which states that "A robot may not harm humanity, or, by inaction, allow humanity to come to harm."
This law was not included in the original three laws, but was later added in response to criticism that the laws did not adequately address the potential dangers of artificial intelligence.\n\n
The laws have been referenced and discussed in a variety of contexts, including science fiction, philosophy, and artificial intelligence.
They have also been the subject of several academic studies, which have examined their philosophical and ethical'
