

TITLE
legacy finetune with t5 issues

COMMENTS
9

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
Hi @stas00 

Splitting of from https://github.com/huggingface/transformers/issues/8771#issuecomment-884865133

There is a lot of great information in your post; thanks for being thorough!

I guess I dont understand what parameters I need to change within the deepspeed config file to properly offload into cpu memory. I have 473 gb of RAM available for offloading, which seems to be enough based on what you listed. I am also using the finetune script in the seq2seq legacy folder. The command is:

`export BS=2; rm -rf output_dir; PYTHONPATH=../../src USE_TF=0 deepspeed --num_gpus=8 ./finetune_trainer.py --model_name_or_path "Rostlab/prot_t5_xl_uniref50" --output_dir output_dir --adam_eps 1e-06 --data_dir /mnt/data --do_eval --do_predict --do_train --evaluation_strategy=steps --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 512 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS --predict_with_generate --eval_steps 25000 --sortish_sampler --test_max_target_length 128 --val_max_target_length 128 --warmup_steps 5 --n_train 60 --n_val 10 --n_test 10 --deepspeed ../../../tests/deepspeed/ds_config_zero3.json --fp16`

I had to modify finetune to include the T5Tokenizer as the AutoTokenizer wouldnt work.

For zero 3 optimization, I am using lower values for stage3_params, since the documentation indicated to use lower values to offload memory.

```
    "zero_optimization": {
        "stage": 3,
        "cpu_offload": true,
        "cpu_offload_params": true,
        "cpu_offload_use_pin_memory" : true,
        "overlap_comm": true,
        "contiguous_gradients": true,
        "stage3_max_live_parameters": 1e3,
        "stage3_max_reuse_distance": 1e3,
        "stage3_prefetch_bucket_size": 2e3,
        "stage3_param_persitance_threshold": 1e3,
        "reduce_bucket_size": 3e3,
        "prefetch_bucket_size": 3e3,
        "sub_group_size": 1e3
    },
```


