Pretrained models for the paper Scaling up Masked Diffusion Models on Text
Scaling law experiments: We provided all pre-trained models in the ar_safetensors and mdm_safetensors folders.
For instance, the checkpoint mdm-1028M-1600e18.safetensors represents an MDM model with 1,028 million non-embedding
parameters and 1,600e18 training FLOPs. Similarly, the checkpoint mdm-170M-100e18-rsl-0.01.safetensors indicates
an MDM model with 170 million non-embedding parameters, 100e18 training FLOPs, and 1% of the dataset subjected
to random sequence lengths during pretraining.
Math reasoning: please see the gsm8k_safetensors folder.
Conditional generation: please see the sharegpt_safetensors folder.
Reverse curse: please see the reverse_safetensors folder
For all models, we provide models in .pth and .safetensors formats.