Pre-training Small Base LMs with Fewer Tokens
Paper
•
2404.08634
•
Published
•
36
•
4
Sunny Sanyal
Sunny111
AI & ML interests
Efficient Training Recipes of Large Models (mostly LLMs)
Recent Activity
replied to
their
post
7 days ago
Are you familiar with reverse residual connections or looping in language models?
Excited to share my Looped-GPT blog post and codebase 🚀
https://github.com/sanyalsunny111/Looped-GPT
TL;DR: looping during pre-training improves generalization.
Plot shows GPT2 LMs pre-trained with 15.73B OWT tokens
P.S. This is my first post here — I have ~4 followers and zero expectations for reach 😄
posted
an
update
10 days ago
Are you familiar with reverse residual connections or looping in language models?
Excited to share my Looped-GPT blog post and codebase 🚀
https://github.com/sanyalsunny111/Looped-GPT
TL;DR: looping during pre-training improves generalization.
Plot shows GPT2 LMs pre-trained with 15.73B OWT tokens
P.S. This is my first post here — I have ~4 followers and zero expectations for reach 😄
upvoted
a
paper
about 1 month ago
Pre-training Small Base LMs with Fewer Tokens
Organizations
commented
a
paper
over 1 year ago
commented
a
paper
almost 2 years ago