

TITLE
TrainingArguments --learning_rate should not be used to set both "lr" and "warmup_max_lr" in DeepSpeed

COMMENTS
4

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
According to this document: https://huggingface.co/docs/transformers/v4.16.2/en/main_classes/deepspeed#shared-configuration, 
--learning_rate is used to set two parameters in DeepSpeed configuration: "lr" and "warmup_max_lr". This causes learning rate never changes, because the initial value is already the maximal value. 

Here is my DeepSpeed configuration:

```
  "optimizer": {
    "type": "AdamW",
    "params": {
      "lr": "auto",
      "betas": "auto"
    }
  },

  "scheduler": {
    "type": "WarmupLR",
    "params": {
      "warmup_min_lr": "auto",
      "warmup_max_lr": "auto" ,
      "warmup_num_steps": "auto"
    }
  },

```

When running with this DeepSpeed configuration, I saw both "lr" and "warmup_max_lr" are set to 5e-5:

![image](https://user-images.githubusercontent.com/51274745/152467841-4c06b0bd-c882-4989-8370-60d09d3f70e3.png)

Unfortunately, the learning rate stayed at 5e-5 all the way because "lr" is bounded by "warmup_max_lr". When not using DeepSpeed, HF is in charge of learning rate, and I see learning rate changes.

I think --learning_rate should be used to set "warmup_min_lr", not the max.