--- library_name: transformers tags: - bitnet - falcon3 base_model: tiiuae/Falcon3-1B-Instruct license: other license_name: falcon-llm-license license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html --- ![image/png](https://cdn-uploads.huggingface.co/production/uploads/62441d1d9fdefb55a0b7d12c/c-tosr0FvMlKuKQTojx_6.png) # Table of Contents 0. [TL;DR](#TL;DR) 1. [Model Details](#model-details) 2. [Training Details](#training-details) 3. [Usage](#usage) 4. [Evaluation](#evaluation) 5. [Citation](#citation) # TL;DR # Model Details ## Model Description - **Developed by:** [https://www.tii.ae](https://www.tii.ae) - **Model type:** Causal decoder-only - instruct / chat version - **Architecture:** Pure-transformer - 1.58bit version - **Language(s) (NLP):** Mainly English - **License:** TII Falcon License 2.0 # Training details The model has been trained following the training strategies from the recent [1-bit LLM HF blogpost](https://huggingface.co/blog/1_58_llm_extreme_quantization) and [1-bit LLM paper](https://github.com/microsoft/unilm/blob/master/bitnet/The-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ.pdf). For more details about the training protocol of this model, please refer to the Falcon-3 technical report, section *Compression*. # Usage Currently to use this model you can either rely on Hugging Face transformers library or [BitNet](https://github.com/microsoft/BitNet) library. You can also play with the model using the [falcon-1.58bit playground](https://huggingface.co/spaces/tiiuae/falcon3-1.58bit-playground) (only for the 7B instruct version). ## 🤗 transformers ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "tiiuae/Falcon3-1B-Instruct-1.58bit" model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, ).to("cuda") # Perform text generation ``` ## BitNet ``` git clone https://github.com/microsoft/BitNet && cd BitNet pip install -r requirements.txt python setup_env.py --hf-repo tiiuae/Falcon3-1B-Instruct-1.58bit -q i2_s python run_inference.py -m models/Falcon3-1B-1.58bit/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv ``` # Evaluation We report in the following table our internal pipeline benchmarks: **Note evaluation results are normalized score from v2 leaderboard tasks - reported results of original models in the blogpost are raw scores**
Benchmark | Llama3-8B-1.58-100B-tokens | Falcon3-1B-Instruct-1.58bit |
---|---|---|
IFEval | 17.91 | 44.5 |
MUSR | 4.87 | 2.78 |
GPQA | 1.83 | 0 |
BBH | 5.36 | 2.24 |
MMLU-PRO | 2.78 | 1.93 |
MATH | 0.26 | 0.17 |
Average | 5.5 | 8.6 |