tiiuae

/

falcon-40b-instruct

like 1.17k

Follow

Technology Innovation Institute 934

I am using the following code, provided in this space, to try to load falcon-40b-instruct on google colab:

import transformers
import torch

model = "tiiuae/falcon-40b-instruct"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)
sequences = pipeline(
   "Girafatron is obsessed with giraffes, the most glorious animal on the face of this Earth. Giraftron believes all other animals are irrelevant when compared to the glorious majesty of the giraffe.\nDaniel: Hello, Girafatron!\nGirafatron:",
    max_length=200,
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")```
 but get the following error:
ValueError: Could not load model tiiuae/falcon-40b-instruct with any of the following classes: (<class 
'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>, <class 
'transformers.models.auto.modeling_tf_auto.TFAutoModelForCausalLM'>).

Previously I was also getting this error:
NameError: name 'init_empty_weights' is not defined
(something must have changed when restarting the kernel)

On Colab, I have the following versions:
torch 2.0.1+cu118
transformers 4.29.2