Qwen2.5
Collection
Qwen2.5 language models, including pretrained and instruction-tuned models of 7 sizes, including 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B.
•
45 items
•
Updated
•
392
yushun zhang
yushun0410
AI & ML interests
LLMs
Recent Activity
upvoted
a
collection
9 days ago
Qwen2.5
upvoted
a
collection
9 days ago
Qwen2.5-Math
Organizations
None yet
yushun0410's activity
upvoted
2
collections
9 days ago
Reacted to
their
post with 🔥🚀👍
5 months ago
Post
4609
Hi Huggingfacers!
Thrilled to introduce Adam-mini, an optimizer that achieves on-par or better performance than AdamW with 45% to 50% less memory footprint. Adam-mini can also achieve 49.5% higher throughput than AdamW on Llama2-7B pre-training.
The design of Adam-mini is inspired by certain Hessian structures we observed on Transformers.
Feel free to try it out! Try switching to Adam-mini with the same hyperparams of AdamW, it would work with only half memory. Hope Adam-mini can help save time, cost, and energy in your tasks!
Paper: "Adam-mini: Use Fewer Learning Rates To Gain More" https://arxiv.org/abs/2406.16793
Code: https://github.com/zyushun/Adam-mini
Thrilled to introduce Adam-mini, an optimizer that achieves on-par or better performance than AdamW with 45% to 50% less memory footprint. Adam-mini can also achieve 49.5% higher throughput than AdamW on Llama2-7B pre-training.
The design of Adam-mini is inspired by certain Hessian structures we observed on Transformers.
Feel free to try it out! Try switching to Adam-mini with the same hyperparams of AdamW, it would work with only half memory. Hope Adam-mini can help save time, cost, and energy in your tasks!
Paper: "Adam-mini: Use Fewer Learning Rates To Gain More" https://arxiv.org/abs/2406.16793
Code: https://github.com/zyushun/Adam-mini
- 1 reply
posted
an
update
5 months ago
Post
4609
Hi Huggingfacers!
Thrilled to introduce Adam-mini, an optimizer that achieves on-par or better performance than AdamW with 45% to 50% less memory footprint. Adam-mini can also achieve 49.5% higher throughput than AdamW on Llama2-7B pre-training.
The design of Adam-mini is inspired by certain Hessian structures we observed on Transformers.
Feel free to try it out! Try switching to Adam-mini with the same hyperparams of AdamW, it would work with only half memory. Hope Adam-mini can help save time, cost, and energy in your tasks!
Paper: "Adam-mini: Use Fewer Learning Rates To Gain More" https://arxiv.org/abs/2406.16793
Code: https://github.com/zyushun/Adam-mini
Thrilled to introduce Adam-mini, an optimizer that achieves on-par or better performance than AdamW with 45% to 50% less memory footprint. Adam-mini can also achieve 49.5% higher throughput than AdamW on Llama2-7B pre-training.
The design of Adam-mini is inspired by certain Hessian structures we observed on Transformers.
Feel free to try it out! Try switching to Adam-mini with the same hyperparams of AdamW, it would work with only half memory. Hope Adam-mini can help save time, cost, and energy in your tasks!
Paper: "Adam-mini: Use Fewer Learning Rates To Gain More" https://arxiv.org/abs/2406.16793
Code: https://github.com/zyushun/Adam-mini
-
- 1 reply
commented
a
paper
5 months ago
authored
2
papers
5 months ago
commented
a
paper
5 months ago
upvoted
a
paper
5 months ago
[FEEDBACK] Daily Papers
95
#32 opened 6 months ago
by
kramp
kramp
commented
a
paper
5 months ago
authored
a
paper
5 months ago