|
--- |
|
|
|
base_model: utter-project/EuroLLM-9B-Instruct |
|
|
|
--- |
|
This is a quantization of the [EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct). |
|
|
|
The EuroLLM project has the goal of creating a suite of LLMs capable of understanding and generating text in all European Union languages as well as some additional relevant languages. EuroLLM-9B is a 9B parameter model trained on 4 trillion tokens divided across the considered languages and several data sources: Web data, parallel data (en-xx and xx-en), and high-quality datasets. EuroLLM-9B-Instruct was further instruction tuned on EuroBlocks, an instruction tuning dataset with focus on general instruction-following and machine translation. |
|
## Evaluations |
|
This model provides an accuracy recovery of 99.61%. |
|
|
|
| __English__ | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__ | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__ | |
|
|:--------------|:--------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------| |
|
| Avg. | 66.35 | 65.35 | |
|
| Arc | 63.3 | 61.7 | |
|
| Hellaswag | 69.4 | 69.0 | |
|
| | | | |
|
| __French__ | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__ | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__ | |
|
| Avg. | 61.67 | 61.3 | |
|
| Arc | 58.1 | 57.3 | |
|
| Hellaswag | 70.2 | 70.3 | |
|
| MMLU | 56.7 | 56.3 | |
|
| | | | |
|
| __German__ | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__ | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__ | |
|
| Avg. | 60.0 | 60.37 | |
|
| Arc | 57.2 | 56.7 | |
|
| Hellaswag | 66.3 | 67.1 | |
|
| MMLU | 56.5 | 57.3 | |
|
| | | | |
|
| __Italian__ | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__ | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__ | |
|
| Avg. | 61.8 | 61.7 | |
|
| Arc | 58.3 | 58.2 | |
|
| Hellaswag | 69.9 | 69.4 | |
|
| MMLU | 57.2 | 57.5 | |
|
| | | | |
|
| __Portuguese__ | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__ | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__ | |
|
| Avg. | 61.47 | 61.37 | |
|
| Arc | 59.1 | 59.3 | |
|
| Hellaswag | 70.3 | 70.2 | |
|
| MMLU | 55.0 | 54.6 | |
|
| | | | |
|
| __Spanish__ | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__ | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__ | |
|
| Avg. | 62.03 | 61.53 | |
|
| Arc | 59.7 | 59.3 | |
|
| Hellaswag | 71.4 | 71 | |
|
| MMLU | 55 | 54.3 | |
|
|
|
We did not check for data contamination. |
|
Evaluation was done using [Eval. Harness](https://github.com/EleutherAI/lm-evaluation-harness) with `limit=1000`. |
|
|
|
## Usage |
|
Install **vLLM** and |
|
run the [server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#openai-compatible-server): |
|
|
|
``` |
|
python -m vllm.entrypoints.openai.api_server --model cortecs/EuroLLM-9B-Instruct-FP8-Dynamic --gpu-memory-util 0.95 |
|
``` |
|
Access the model: |
|
``` |
|
curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d ' { |
|
"model": "cortecs/EuroLLM-9B-Instruct-FP8-Dynamic", |
|
"prompt": "San Francisco is a" |
|
} ' |
|
``` |
|
⚡ This model is optimized to handle heavy workloads providing a total throughput of ️**1976 tokens per second** using one NVIDIA L4 ⚡ |
