TรLU 3: Pushing Frontiers in Open Language Model Post-Training
Paper
โข
2411.15124
โข
Published
โข
14
Quentin Tardif
ntnq
AI & ML interests
None yet
Recent Activity
upvoted
a
paper
about 3 hours ago
TรLU 3: Pushing Frontiers in Open Language Model Post-Training
liked
a Space
18 days ago
HuggingFaceFW/blogpost-fine-tasks
upvoted
a
paper
25 days ago
CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation
Generation
Organizations
ntnq's activity
upvoted
a
paper
about 3 hours ago
liked
a
Space
18 days ago
upvoted
a
paper
25 days ago
liked
2
Spaces
25 days ago
liked
a
dataset
28 days ago
liked
2
models
about 1 month ago
upvoted
a
collection
about 1 month ago
liked
a
Space
about 1 month ago
Reacted to
Taylor658's
post with ๐
about 1 month ago
Post
2215
The Mystery Bot ๐ต๏ธโโ๏ธ saga I posted about from earlier this week has been solved...๐ค
Cohere for AI has just announced its open source Aya Expanse multilingual model. The Initial release supports 23 languages with more on the way soon.๐ ๐
You can also try Aya Expanse via SMS on your mobile phone using the global WhatsApp number or one of the initial set of country specific numbers listed below.โฌ๏ธ
๐WhatsApp - +14313028498
Germany - (+49) 1771786365
USA โ +18332746219
United Kingdom โ (+44) 7418373332
Canada โ (+1) 2044107115
Netherlands โ (+31) 97006520757
Brazil โ (+55) 11950110169
Portugal โ (+351) 923249773
Italy โ (+39) 3399950813
Poland - (+48) 459050281
Cohere for AI has just announced its open source Aya Expanse multilingual model. The Initial release supports 23 languages with more on the way soon.๐ ๐
You can also try Aya Expanse via SMS on your mobile phone using the global WhatsApp number or one of the initial set of country specific numbers listed below.โฌ๏ธ
๐WhatsApp - +14313028498
Germany - (+49) 1771786365
USA โ +18332746219
United Kingdom โ (+44) 7418373332
Canada โ (+1) 2044107115
Netherlands โ (+31) 97006520757
Brazil โ (+55) 11950110169
Portugal โ (+351) 923249773
Italy โ (+39) 3399950813
Poland - (+48) 459050281
- 1 reply
liked
a
Space
about 1 month ago
liked
a
model
about 1 month ago
Reacted to
m-ric's
post with ๐
about 1 month ago
Post
3035
๐ ๐๐ฅ๐-๐ฌ๐๐ก๐จ๐จ๐ฅ ๐๐๐๐ฌ ๐๐๐ง ๐๐๐ญ๐ฎ๐๐ฅ๐ฅ๐ฒ ๐ซ๐ข๐ฏ๐๐ฅ ๐๐๐ง๐๐ฒ ๐ญ๐ซ๐๐ง๐ฌ๐๐จ๐ซ๐ฆ๐๐ซ๐ฌ!
Researchers from Mila and Borealis AI just have shown that simplified versions of good old Recurrent Neural Networks (RNNs) can match the performance of today's transformers.
They took a fresh look at LSTMs (from 1997!) and GRUs (from 2014). They stripped these models down to their bare essentials, creating "minLSTM" and "minGRU". The key changes:
โถ Removed dependencies on previous hidden states in the gates
โท Dropped the tanh that had been added to restrict output range in order to avoid vanishing gradients
โธ Ensured outputs are time-independent in scale (not sure I understood that well either, don't worry)
โก๏ธ As a result, you can use a โparallel scanโ algorithm to train these new, minimal RNNs, in parallel, taking 88% more memory but also making them 200x faster than their traditional counterparts for long sequences
๐ฅ The results are mind-blowing! Performance-wise, they go toe-to-toe with Transformers or Mamba.
And for Language Modeling, they need 2.5x fewer training steps than Transformers to reach the same performance! ๐
๐ค Why does this matter?
By showing there are simpler models with similar performance to transformers, this challenges the narrative that we need advanced architectures for better performance!
๐ฌย Franรงois Chollet wrote in a tweet about this paper:
โThe fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't fundamentally important in the curve-fitting paradigm (aka deep learning)โ
โCurve-fitting is about embedding a dataset on a curve. The critical factor is the dataset, not the specific hard-coded bells and whistles that constrain the curve's shape.โ
Itโs the Bitter lesson by Rich Sutton striking again: donโt need fancy thinking architectures, just scale up your model and data!
Read the paper ๐ย Were RNNs All We Needed? (2410.01201)
Researchers from Mila and Borealis AI just have shown that simplified versions of good old Recurrent Neural Networks (RNNs) can match the performance of today's transformers.
They took a fresh look at LSTMs (from 1997!) and GRUs (from 2014). They stripped these models down to their bare essentials, creating "minLSTM" and "minGRU". The key changes:
โถ Removed dependencies on previous hidden states in the gates
โท Dropped the tanh that had been added to restrict output range in order to avoid vanishing gradients
โธ Ensured outputs are time-independent in scale (not sure I understood that well either, don't worry)
โก๏ธ As a result, you can use a โparallel scanโ algorithm to train these new, minimal RNNs, in parallel, taking 88% more memory but also making them 200x faster than their traditional counterparts for long sequences
๐ฅ The results are mind-blowing! Performance-wise, they go toe-to-toe with Transformers or Mamba.
And for Language Modeling, they need 2.5x fewer training steps than Transformers to reach the same performance! ๐
๐ค Why does this matter?
By showing there are simpler models with similar performance to transformers, this challenges the narrative that we need advanced architectures for better performance!
๐ฌย Franรงois Chollet wrote in a tweet about this paper:
โThe fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't fundamentally important in the curve-fitting paradigm (aka deep learning)โ
โCurve-fitting is about embedding a dataset on a curve. The critical factor is the dataset, not the specific hard-coded bells and whistles that constrain the curve's shape.โ
Itโs the Bitter lesson by Rich Sutton striking again: donโt need fancy thinking architectures, just scale up your model and data!
Read the paper ๐ย Were RNNs All We Needed? (2410.01201)
- 2 replies
upvoted
2
papers
about 1 month ago
liked
a
dataset
about 2 months ago
liked
a
Space
about 2 months ago
liked
2
models
about 2 months ago
