WebOrganizer/LM-1b_1x-DCLMFasttext_over_Topics_x_Formats_for_MMLU_and_Hellaswag

[Paper] [Website] [GitHub]

A 1.4B parameter model trained for 29B tokens from WebOrganizer/Corpus-200B.

The training data for this model was selected via:

1. Selection method: Top scores from DCLM-Fasttext Model
2. Domain definition: Combination of 24 WebOrganizer Topics and 24 WebOrganizer Formats
3. Domain mixture: MMLU and HellaSwag

Repository Contents

Besides the HuggingFace model and tokenizer, the repository contains:

• open_lm/: Contains the OpenLM config and final checkpoint
• evals/: Evaluation results for various benchmarks
  • core_9mcqa/: Results of 9 multiple choice QA tasks with the OLMES evaluation framework
  • mmlu/: MMLU results with the OLMES evaluation framework
  • dclm/: Results using the DCLM evaluation framework
  • perplexity/: Perplexity results using the huggingface trainer
• indices.tar.zst: The indices for the selected documents in each shard of the Corpus-200B dataset used for training. The indices can be extracted with tar --use-compress-program "zstd" -xf indices.tar.zst.

Usage

To use this model, you need to install the open_lm library and add from open_lm.hf import * before loading the model with AutoModel.from_pretrained(...).

Citation

@article{wettig2025organize,
  title={Organize the Web: Constructing Domains Enhances Pre-Training Data Curation},
  author={Alexander Wettig and Kyle Lo and Sewon Min and Hannaneh Hajishirzi and Danqi Chen and Luca Soldaini},
  journal={arXiv preprint arXiv:2502.10341},
  year={2025}
}