Edit model card

Lil-Bevo-X

Lil-Bevo-X is UT Austin's submission to the BabyLM challenge, specifically the strict track.

Link to GitHub Repo

Model training regime:

  1. 5 epochs on MAESTRO dataset (85M non-language music tokens) combined with strict small dataset.
  2. 50 epochs of pretraining with sequence length of 128 on strict dataset.
  3. 150 epochs of pretraining with sequence length of 512 on strict dataset.
  4. 10 epochs of targeted MLM.

This README will be updated with more details soon.

Downloads last month
10
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Collection including venkatasg/lil-bevo-x