Note Original dataset used to train tokenisers and models.

pietrolesci/tokenisers

Updated 15 days ago

Note Tokenisers trained on the MiniPile. The `_raw_tokenisers` folder contains the original tokenisers trained with a vocabulary size of 320k. Then, each folder is a `transformers`-compatible tokeniser of a smaller size.

Note Tokenised MiniPile dataset(s). Each split correponds to a tokeniser in `pietrolesci/tokenisers`.

pietrolesci/me57M-tied_minipile_bpe8064minipile

Updated about 6 hours ago

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me57M-tied_minipile_bpe32000minipile

Updated about 6 hours ago

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me57M-tied_minipile_bpe128000minipile

Updated about 6 hours ago

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me57M-tied_minipile_wordpiece32000minipile

Updated about 6 hours ago

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me57M-tied_minipile_bpe2wp32000minipile

Updated about 6 hours ago

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved every 2k steps. The bpe2wp nomenclature means that we choose the merges using the BPE objective, and we tokenised the MiniPile using the resulting vocabulary and the WordPiece tokenisation function (i.e., longest prefix match).

pietrolesci/me340M-tied_minipile_bpe32000minipile

Updated about 6 hours ago • 56

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

pietrolesci/me850M_minipile_bpe32000minipile

Updated about 5 hours ago • 54

Note Model trained for 50k steps on the MiniPile dataset. Each branch is a different checkpoint saved each 2k steps.

Note Log-probabilities computed on the validation set of the MiniPile dataset using the models above.