tiiuae

/

falcon-mamba-7b-instruct

like 64

Follow

Technology Innovation Institute 931

When attempting to run the sample code using transformers-4.44.0, huggingface-hub-0.24.5, tokenizers-0.19.1:

# pip install accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-mamba-7b-instruct")
model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-mamba-7b-instruct", device_map="auto")
# We use the tokenizer's chat template to format each message - see https://huggingface.co/docs/transformers/main/en/chat_templating
messages = [
{"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")
outputs = model.generate(input_ids, max_new_tokens=30)
print(tokenizer.decode(outputs[0]))

I get the following output:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
File c:\WBG\Anaconda3\envs\cuda\Lib\site-packages\transformers\models\auto\configuration_auto.py:993, in AutoConfig.from_pretrained(cls,
pretrained_model_name_or_path, **kwargs)
992 try:
--> 993 config_class = CONFIG_MAPPING[config_dict["model_type"]]
994 except KeyError:

File c:\WBG\Anaconda3\envs\cuda\Lib\site-packages\transformers\models\auto\configuration_auto.py:695, in _LazyConfigMapping.__getitem__(self,
key)
694 if key not in self._mapping:
--> 695 raise KeyError(key)
696 value = self._mapping[key]

KeyError: 'falcon_mamba'

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)
Cell In[1], line 5
2 from transformers import AutoTokenizer, AutoModelForCausalLM
4 tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-mamba-7b-instruct")
----> 5 model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-mamba-7b-instruct", device_map="auto")
7 # We use the tokenizer's chat template to format each message - see https://huggingface.co/docs/transformers/main/en/chat_templating
8 messages = [
9 {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
10 ]


File c:\WBG\Anaconda3\envs\cuda\Lib\site-packages\transformers\models\auto\auto_factory.py:524, in _BaseAutoModelClass.from_pretrained(cls,
pretrained_model_name_or_path, *model_args, **kwargs)

521 if kwargs.get("quantization_config", None) is not None:

522 _ = kwargs.pop("quantization_config")

--> 524 config, kwargs = AutoConfig.from_pretrained(

525 pretrained_model_name_or_path,

526 return_unused_kwargs=True,

527 trust_remote_code=trust_remote_code,

528 code_revision=code_revision,

529 _commit_hash=commit_hash,

530 **hub_kwargs,

531 **kwargs,

532 )

534 # if torch_dtype=auto was passed here, ensure to pass it on

535 if kwargs_orig.get("torch_dtype", None) == "auto":


File c:\WBG\Anaconda3\envs\cuda\Lib\site-packages\transformers\models\auto\configuration_auto.py:995, in AutoConfig.from_pretrained(cls,
pretrained_model_name_or_path, **kwargs)

993 config_class = CONFIG_MAPPING[config_dict["model_type"]]

994 except KeyError:

--> 995 raise ValueError(

996 f"The checkpoint you are trying to load has model type {config_dict['model_type']} "

997 "but Transformers does not recognize this architecture. This could be because of an "

998 "issue with the checkpoint, or because your version of Transformers is out of date."

999 )

1000 return config_class.from_dict(config_dict, **unused_kwargs)

1001 else:

1002 # Fallback: use pattern matching on the string.

1003 # We go from longer names to shorter names to catch roberta before bert (for instance)



ValueError: The checkpoint you are trying to load has model type falcon_mamba but Transformers does not recognize this architecture. This could be
because of an issue with the checkpoint, or because your version of Transformers is out of date.


Can you please help?