Tried and tested mixes for strong pretraining. Inspired by https://huggingface.co/blog/codelion/optimal-dataset-mixing
AI & ML interests
We release large pre-training datasets to accelerate open LLM development. Part of the Hugging Face Science team (hf.co/science)
Recent Activity
View all activity
Papers
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
-
FineWeb: decanting the web for the finest text data at scale
🍷1.33kExplore and download the FineWeb web‑text dataset
-
HuggingFaceFW/fineweb
Viewer • Updated • 52.5B • 649k • 2.77k -
HuggingFaceFW/fineweb-edu
Viewer • Updated • 3.5B • 355k • 1.04k -
HuggingFaceFW/fineweb-edu-score-2
Viewer • Updated • 13.9B • 18.4k • 85
1.8B models trained on 350BT to compare different pretraining datasets
-
HuggingFaceFW/ablation-model-fineweb-edu
Text Generation • 2B • Updated • 404 • 23 -
HuggingFaceFW/ablation-model-fineweb-v1
Text Generation • 2B • Updated • 267 • 14 -
HuggingFaceFW/ablation-model-refinedweb
Text Generation • 2B • Updated • 19 • 3 -
HuggingFaceFW/ablation-model-c4
Text Generation • 2B • Updated • 12 • 4
-
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
Paper • 2506.20920 • Published • 78 -
HuggingFaceFW/fineweb-2
Viewer • Updated • 4.48B • 118k • 784 -
Scaling FineWeb to 1000+ languages: Step 1: finding signal in 100s of evaluation tasks
📝91Evaluate multilingual models using FineTasks
FineWeb-Edu datasets, classifier and ablation model
-
HuggingFaceFW/fineweb-edu
Viewer • Updated • 3.5B • 355k • 1.04k -
HuggingFaceFW/fineweb-edu-score-2
Viewer • Updated • 13.9B • 18.4k • 85 -
HuggingFaceFW/fineweb-edu-classifier
Text Classification • 0.1B • Updated • 35.1k • • 211 -
HuggingFaceFW/ablation-model-fineweb-edu
Text Generation • 2B • Updated • 404 • 23
Ablation models trained for our data experiments.
-
HuggingFaceFW/ablation-exp-textext-warc_trafilatura-28BT
Text Generation • 2B • Updated • 7 • 1 -
HuggingFaceFW/ablation-exp-textext-wet-28BT
Text Generation • 2B • Updated • 7 -
HuggingFaceFW/ablation-exp-fw-base_filtering-350BT
Text Generation • 2B • Updated • 6 -
HuggingFaceFW/ablation-exp-dedup-global_minhash-350BT
Text Generation • 2B • Updated • 4
Tried and tested mixes for strong pretraining. Inspired by https://huggingface.co/blog/codelion/optimal-dataset-mixing
-
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
Paper • 2506.20920 • Published • 78 -
HuggingFaceFW/fineweb-2
Viewer • Updated • 4.48B • 118k • 784 -
Scaling FineWeb to 1000+ languages: Step 1: finding signal in 100s of evaluation tasks
📝91Evaluate multilingual models using FineTasks
-
FineWeb: decanting the web for the finest text data at scale
🍷1.33kExplore and download the FineWeb web‑text dataset
-
HuggingFaceFW/fineweb
Viewer • Updated • 52.5B • 649k • 2.77k -
HuggingFaceFW/fineweb-edu
Viewer • Updated • 3.5B • 355k • 1.04k -
HuggingFaceFW/fineweb-edu-score-2
Viewer • Updated • 13.9B • 18.4k • 85
FineWeb-Edu datasets, classifier and ablation model
-
HuggingFaceFW/fineweb-edu
Viewer • Updated • 3.5B • 355k • 1.04k -
HuggingFaceFW/fineweb-edu-score-2
Viewer • Updated • 13.9B • 18.4k • 85 -
HuggingFaceFW/fineweb-edu-classifier
Text Classification • 0.1B • Updated • 35.1k • • 211 -
HuggingFaceFW/ablation-model-fineweb-edu
Text Generation • 2B • Updated • 404 • 23
1.8B models trained on 350BT to compare different pretraining datasets
-
HuggingFaceFW/ablation-model-fineweb-edu
Text Generation • 2B • Updated • 404 • 23 -
HuggingFaceFW/ablation-model-fineweb-v1
Text Generation • 2B • Updated • 267 • 14 -
HuggingFaceFW/ablation-model-refinedweb
Text Generation • 2B • Updated • 19 • 3 -
HuggingFaceFW/ablation-model-c4
Text Generation • 2B • Updated • 12 • 4
Ablation models trained for our data experiments.
-
HuggingFaceFW/ablation-exp-textext-warc_trafilatura-28BT
Text Generation • 2B • Updated • 7 • 1 -
HuggingFaceFW/ablation-exp-textext-wet-28BT
Text Generation • 2B • Updated • 7 -
HuggingFaceFW/ablation-exp-fw-base_filtering-350BT
Text Generation • 2B • Updated • 6 -
HuggingFaceFW/ablation-exp-dedup-global_minhash-350BT
Text Generation • 2B • Updated • 4