|
--- |
|
|
|
license: other |
|
license_link: https://llama.meta.com/llama3/license/ |
|
base_model: meta-llama/Llama-3.3-70B-Instruct |
|
|
|
--- |
|
This is a quantization of the [Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama--Llama-3.3-70B-Instruct). |
|
|
|
The Meta Llama 3.3 is a state-of-the-art multilingual large language model (LLM) with 70 billion parameters, pretrained and instruction-tuned for exceptional performance in generative text-based tasks. Optimized for multilingual dialogue, it supports English and seven additional languages: French, German, Hindi, Italian, Portuguese, Spanish, and Thai, enabling seamless communication across diverse audiences. The model consistently outperforms both open-source and proprietary chat models on key industry benchmarks, delivering superior quality, safety, and helpfulness. Its advanced features and multilingual support position Llama 3.3 as a powerful tool for building innovative AI applications. |
|
## Evaluations |
|
This model provides an accuracy recovery of 99.67%. |
|
|
|
| __English__ | __[Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama--Llama-3.3-70B-Instruct)__ | __[Llama-3.3-70B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs--Llama-3.3-70B-Instruct-FP8-Dynamic)__ | |
|
|:--------------|:------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------| |
|
| Avg. | 74.1 | 73.75 | |
|
| Arc | 71.7 | 71.6 | |
|
| Hellaswag | 76.5 | 75.9 | |
|
| | | | |
|
| __French__ | __[Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama--Llama-3.3-70B-Instruct)__ | __[Llama-3.3-70B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs--Llama-3.3-70B-Instruct-FP8-Dynamic)__ | |
|
| Avg. | 73.07 | 72.87 | |
|
| Arc | 64.7 | 64.5 | |
|
| Hellaswag | 76.6 | 76.6 | |
|
| MMLU | 77.9 | 77.5 | |
|
| | | | |
|
| __German__ | __[Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama--Llama-3.3-70B-Instruct)__ | __[Llama-3.3-70B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs--Llama-3.3-70B-Instruct-FP8-Dynamic)__ | |
|
| Avg. | 70.07 | 69.83 | |
|
| Arc | 61.8 | 61.2 | |
|
| Hellaswag | 71.2 | 71.1 | |
|
| MMLU | 77.2 | 77.2 | |
|
| | | | |
|
| __Italian__ | __[Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama--Llama-3.3-70B-Instruct)__ | __[Llama-3.3-70B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs--Llama-3.3-70B-Instruct-FP8-Dynamic)__ | |
|
| Avg. | 73.67 | 73.37 | |
|
| Arc | 66.5 | 65.7 | |
|
| Hellaswag | 76.0 | 76.2 | |
|
| MMLU | 78.5 | 78.2 | |
|
| | | | |
|
| __Portuguese__ | __[Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama--Llama-3.3-70B-Instruct)__ | __[Llama-3.3-70B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs--Llama-3.3-70B-Instruct-FP8-Dynamic)__ | |
|
| Avg. | 74.4 | 73.87 | |
|
| Arc | 66.4 | 65.5 | |
|
| Hellaswag | 77.2 | 76.9 | |
|
| MMLU | 79.6 | 79.2 | |
|
| | | | |
|
| __Spanish__ | __[Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama--Llama-3.3-70B-Instruct)__ | __[Llama-3.3-70B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs--Llama-3.3-70B-Instruct-FP8-Dynamic)__ | |
|
| Avg. | 74 | 74.13 | |
|
| Arc | 65.8 | 65.8 | |
|
| Hellaswag | 77.1 | 77.2 | |
|
| MMLU | 79.1 | 79.4 | |
|
|
|
We did not check for data contamination. |
|
Evaluation was done using [Eval. Harness](https://github.com/EleutherAI/lm-evaluation-harness) with `limit=1000`. |
|
|
|
## Usage |
|
Install **vLLM** and |
|
run the [server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#openai-compatible-server): |
|
|
|
``` |
|
python -m vllm.entrypoints.openai.api_server --model cortecs/Llama-3.3-70B-Instruct-FP8-Dynamic |
|
``` |
|
Access the model: |
|
``` |
|
curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d ' { |
|
"model": "cortecs/Llama-3.3-70B-Instruct-FP8-Dynamic", |
|
"prompt": "San Francisco is a" |
|
} ' |
|
``` |
|
⚡ This model is optimized to handle heavy workloads providing a total throughput of ️**1485 tokens per second** using one NVIDIA H100 ⚡ |
