license: apache-2.0
datasets:
- andreabac3/Quora-Italian-Fauno-Baize
- andreabac3/StackOverflow-Italian-Fauno-Baize
- andreabac3/MedQuaAD-Italian-Fauno-Baize
language:
- it
- en
pipeline_tag: text-generation
cerbero-7b Italian LLM 🚀
🔥 Attention! The new and more capable version of cerbero-7b is now available!
📢 cerbero-7b is the first 100% Free and Open Source Italian Large Language Model (LLM) ready to be used for research or commercial applications.
Try an online demo here (quantized demo running on CPU, a lot less powerful than the original cerbero-7b)
Built on top of mistral-7b, which outperforms Llama2 13B across all benchmarks and surpasses Llama1 34B in numerous metrics.
cerbero-7b is specifically crafted to fill the void in Italy's AI landscape.
A cambrian explosion of Italian Language Models is essential for building advanced AI architectures that can cater to the diverse needs of the population.
cerbero-7b, alongside companions like Camoscio and Fauno, aims to help kick-start this revolution in Italy, ushering in an era where sophisticated AI solutions can seamlessly interact with and understand the intricacies of the Italian language, thereby empowering innovation across industries and fostering a deeper connection between technology and the people it serves.
cerbero-7b is released under the permissive Apache 2.0 license, allowing unrestricted usage, even for commercial applications.
Model Evaluation Results 📈
The cerbero-7b model has been rigorously evaluated across several benchmarks to demonstrate its proficiency in understanding and generating Italian text. Below are the summarized results showcasing its performance:
SQuAD-it Evaluation
The Stanford Question Answering Dataset (SQuAD) in Italian (SQuAD-it) is used to evaluate the model's reading comprehension and question-answering capabilities. The following table presents the F1 score and Exact Match (EM) metrics:
| Model | F1 Score | Exact Match (EM) |
|---|---|---|
| cerbero-7b | 72.55% | 55.6% |
| Fauno | 44.46% | 0.00% |
| Camoscio | 37.42% | 0.00% |
| mistral-7b | 15.55% | 8.50% |
EVALITA Benchmark Results
EVALITA benchmarks assess the model's performance in tasks like toxicity detection, irony detection, and sentiment analysis. The table below shows the F1 scores for these tasks:
| Model | Toxicity Detection | Irony Detection | Sentiment Analysis |
|---|---|---|---|
| cerbero-7b | 63.04% | 48.51% | 61.80% |
| Fauno | 33.84% | 39.17% | 12.23% |
| Camoscio | 38.18% | 39.65% | 13.33% |
| mistral-7b | 34.16% | 34.16% | 12.14% |
Why Cerbero? 🤔
The name "Cerbero," inspired by the three-headed dog that guards the gates of the Underworld in Greek mythology, encapsulates the essence of our model, drawing strength from three foundational pillars:
Base Model: mistral-7b 🏗️ cerbero-7b builds upon the formidable mistral-7b as its base model. This choice ensures a robust foundation, leveraging the power and capabilities of a cutting-edge language model.
Datasets: Cerbero Dataset 📚 The Cerbero Dataset is a groundbreaking collection specifically curated to enhance the proficiency of cerbero-7b in understanding and generating Italian text. This dataset is a product of an innovative method combining dynamic self-chat mechanisms with advanced Large Language Model (LLM) technology. Refer to the paper for more details.
Licensing: Apache 2.0 🕊️ Released under the permissive Apache 2.0 license, cerbero-7b promotes openness and collaboration. This licensing choice empowers developers with the freedom for unrestricted usage, fostering a community-driven approach to advancing AI in Italy and beyond.
Training Details 🚀
cerbero-7b is a fully fine-tuned LLM, distinguishing itself from LORA or QLORA fine-tunes. The model is trained on an expansive Italian Large Language Model (LLM) using synthetic datasets generated through dynamic self-chat on a large context window of 8192 tokens
Dataset Composition 📊
📢 Details on the Cerbero Dataset will be updated shortly!
Training Setup ⚙️
cerbero-7b is trained on an NVIDIA DGX H100:
- Hardware: Utilizing 8xH100 GPUs, each with 80 GB VRAM. 🖥️
- Parallelism: DeepSpeed Zero stage 1 parallelism for optimal training efficiency.✨
The model has been trained for 1 epoch, ensuring a convergence of knowledge and proficiency in handling diverse linguistic tasks.
Getting Started 🚀
You can load cerbero-7b using 🤗transformers
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("galatolo/cerbero-7b")
tokenizer = AutoTokenizer.from_pretrained("galatolo/cerbero-7b")
prompt = """Questa è una conversazione tra un umano ed un assistente AI.
[|Umano|] Come posso distinguere un AI da un umano?
[|Assistente|]"""
input_ids = tokenizer(prompt, return_tensors='pt').input_ids
with torch.no_grad():
output_ids = model.generate(input_ids, max_new_tokens=128)
generated_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(generated_text)
GGUF and llama.cpp
cerbero-7b is fully compatibile with llama.cpp
You can find the original and quantized versions of cerbero-7b in the gguf format here
from llama_cpp import Llama
from huggingface_hub import hf_hub_download
llm = Llama(
model_path=hf_hub_download(
repo_id="galatolo/cerbero-7b-gguf",
filename="ggml-model-Q4_K.gguf",
),
n_ctx=4086,
)
llm.generate("""Questa è una conversazione tra un umano ed un assistente AI.
[|Umano|] Come posso distinguere un AI da un umano?
[|Assistente|]""")