

TITLE
why My Albert pretrain loss can't decrease?

COMMENTS
2

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 📚 Migration

## Information

<!-- Important information -->

Model i am using Albert

Language I am using the model on just digits(Desensitized Chinese)

The problem arises when using: Albert Trainer

## Details
The loss can decrease normal when I take this config to RoBerta, I use Albert replace RoBerta, The loss can't decrease, I don't know what's the problem, please help

```
%%time
bert_file = './albert'
from transformers import Trainer, TrainingArguments
from transformers import LineByLineTextDataset, DataCollatorForLanguageModeling
from transformers import AlbertConfig, AlbertForMaskedLM

config = AlbertConfig(
    hidden_size = 768, 
    num_attention_heads = 12,
    intermediate_size = 3072, 
    vocab_size = vocab_size + 10
)
model = AlbertForMaskedLM(config=config)

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

train_dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="./my_wiki_file",
    block_size=128,
)

training_args = TrainingArguments(
    output_dir=bert_file,
    overwrite_output_dir=True,
    num_train_epochs=40,
    per_device_train_batch_size=64,
    save_steps=100000,
    save_total_limit=2,
    prediction_loss_only=False,
)

%%time
trainer = Trainer(
    model = model,
    args = training_args,
    data_collator = data_collator,
    train_dataset = train_dataset
)
trainer.train()
```
This is result
```
Step	Training Loss
500	6.687300
1000	4.034700
1500	3.826200
2000	3.777200
2500	3.788800
3000	3.751100
3500	3.780000
4000	3.772900
4500	3.795800
5000	3.737000
5500	3.782300
6000	3.775600
6500	3.821400
7000	3.730200
7500	3.751700
8000	3.787000
8500	3.824500
9000	3.746300
9500	3.782600
10000	3.770600
```

## Environment info
- `transformers` version:4.5.0.dev0
- Platform:Kaggle notebook
- Python version:3.7
- PyTorch version (GPU?):
- torch version: 1.8.0
- Using GPU in script?:YES
- Using distributed or parallel set-up in script?:NO


## Checklist

- [ ] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [ ] I checked if a related official extension example runs on my machine.
