metadata
language:
- en
tags:
- falcon3
- falcon3_mamba
base_model:
- tiiuae/Falcon3-Mamba-7B-Base
Falcon3-Mamba-7B-Instruct
Falcon3 family of Open Foundation Models is a set of pretrained and instruct LLMs ranging from 1B to 10B.
This repository contains the Falcon3-Mamba-7B-Instruct. It achieves ,compared to similar SSM-based models of the same size, state of art results (at release's time) on reasoning, language understanding, instruction following, code and mathematics tasks. Falcon3-Mamba-7B-Instruct supports a context length up to 32K and 1 language (english).
Model Details
- Architecture(same as Falcon-Mamba-7b)
- Mamba1 based causal decoder only architecture trained on a causal language modeling task (i.e., predict the next token).
- 64 decoder blocks
- width: 4096
- state_size: 16
- 32k context length
- 65k vocab size
- Pretrained on 7 Teratokens of datasets comprising of web, code, STEM and high quality data using 2048 H100 GPU chips
- Postrained on 1.2 million samples of STEM, conversations, code, and safety.
- Developed by Technology Innovation Institute
- License: TII Falcon-LLM License 2.0
- Model Release Date: December 2024
Getting started
Click to expand
from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "tiiuae/Falcon3-Mamba-7B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
prompt = "How many hours in one day?"
messages = [
{"role": "system", "content": "You are a helpful friendly assistant Falcon3 from TII, try to follow instructions as much as possible."},
{"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
**model_inputs,
max_new_tokens=1024
)
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)
Benchmarks
We report in the following table our internal pipeline benchmarks:
| Category | Benchmark | Zamba2-7B-instruct | Jamba-1.5-Mini | Qwen2-7B-Instruct | Llama-3.1-8B-Instruct | Falcon3-Mamba-7B-Instruct |
|---|---|---|---|---|---|---|
| General | MMLU (5-shot) | - | 68.7% | - | 55.9% | - |
| MMLU-PRO (5-shot) | 32.4% | 31.6% | 31.6% | 21.8% | 26.3% | |
| IFEval | 69.9% | 65.7% | 56.8% | 78.8% | 71.7% | |
| Math | GSM8K (5-shot) | - | 74.9% | - | 19.2% | - |
| MATH Lvl-5 (4-shot) | - | 6.9% | 9.44% | 10.4% | 27.3% | |
| Reasoning | Arc Challenge (25-shot) | - | 54.3% | - | 46.6% | - |
| GPQA (0-shot) | 10.3% | 11.1% | 6.4% | 33.6% | 7.2% | |
| MUSR (0-shot) | 8.2% | 12.2% | 7.4% | 38.6% | 8.3% | |
| BBH (3-shot) | 33.3% | 35.3% | 37.8% | 43.7% | 25.2% | |
| CommonSense Understanding | PIQA (0-shot) | - | 82.3% | - | 78.9% | - |
| SciQ (0-shot) | - | 94.9% | - | 80.2% | - | |
| Winogrande (0-shot) | - | 64.5% | - | - | - | |
| OpenbookQA (0-shot) | - | 34.6% | - | 46.2% | - |
Citation
If Falcon3 family were helpful to your work, feel free to give us a cite.
@misc{Falcon3,
title = {The Falcon 3 family of Open Models},
author = {TII Team},
month = {December},
year = {2024}
}