Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler
Paper
•
2408.13359
•
Published
•
22
Power-LM
Dense & MoE LLMs trained with power learning rate scheduler.
Dense & MoE LLMs trained with power learning rate scheduler.