README.md

---
language:
- en
- fr
- es
- pt
tags:
- falcon3
base_model: tiiuae/Falcon3-10B-Instruct
---

# Falcon3-10B-Instruct-GGUF

Tired of needing massive GPUs just to experiment with the latest Large Language Models?  Wish you could run powerful LLMs locally on your laptop or even your phone? This GGUF model makes it possible!

Falcon3-10B-Instruct-GGUF is designed for efficient inference on consumer-grade hardware.  It leverages the GGUF format for optimal performance, allowing you to experience the power of LLMs without the need for expensive hardware.

Whether you're a student, hobbyist, or developer, this model opens up a world of possibilities for exploring natural language processing, text generation, and AI-powered applications right at your fingertips.


## Getting started

### 1. Download GGUF models from hugging face
First, download the model from Hugging Face. You can use the `huggingface_hub` library or download it manually:

```bash
pip install huggingface_hub
huggingface-cli download {model_name}
```

This will download the model to your current directory.  Make sure to replace {model_name} with the actual username and model name from your Hugging Face repository.

## 2. Install llama.cpp

You have several options for installing llama.cpp:

**1. Build from source:**

This gives you the most flexibility and control. Follow the instructions in the llama.cpp repository to build from source:

   ```bash
   git clone https://github.com/ggerganov/llama.cpp
   cd llama.cpp
   cmake -B build
   cmake --build build --config Release
   ```
For more information about how to build llama.cpp from source please refere to llama.cpp documentation on how to build from source: **[llama.cpp build from source](https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md)**.

**2. Download pre-built binaries:**

If you prefer a quicker setup, you can download pre-built binaries for your operating system. Check the llama.cpp repository for available binaries. 

**3. Use Docker:**

For a more contained environment, you can use the official llama.cpp Docker image. Refer to the llama.cpp documentation for instructions on how to use the Docker image.

For detailed instructions and more information, please check the llama.cpp documentation on docker: **[llama.cpp docker](https://github.com/ggerganov/llama.cpp/blob/master/docs/docker.mdg)**.


### 3. Start playing with your model
- <details open>
    <summary>Run simple text completion</summary>

    ```bash
    llama-cli -m {path-to-gguf-model} -p "I believe the meaning of life is" -n 128
    ```

    </details>

- <details>
    <summary>Run in conversation mode</summary>

    ```bash
    llama-cli -m {path-to-gguf-model} -p "You are a helpful assistant" -cnv -co
    </details>
    ```

# Citation
If Falcon3 family were helpful to your work, feel free to give us a cite.

```
@misc{Falcon3,
    title = {The Falcon 3 family of Open Models},
    author = {TII Team},
    month = {December},
    year = {2024}
}
```