Nanobit commited on
Commit
94d03c8
1 Parent(s): 11ddccb

Clarify pre-tokenize before multigpu (#359)

Browse files
Files changed (1) hide show
  1. README.md +8 -1
README.md CHANGED
@@ -524,7 +524,14 @@ Run
524
  accelerate launch scripts/finetune.py configs/your_config.yml
525
  ```
526
 
527
- #### Multi-GPU Config
 
 
 
 
 
 
 
528
 
529
  - llama FSDP
530
  ```yaml
 
524
  accelerate launch scripts/finetune.py configs/your_config.yml
525
  ```
526
 
527
+ #### Multi-GPU
528
+
529
+ It is recommended to pre-tokenize dataset with the following before finetuning:
530
+ ```bash
531
+ CUDA_VISIBLE_DEVICES="" accelerate ... --prepare_ds_only
532
+ ```
533
+
534
+ ##### Config
535
 
536
  - llama FSDP
537
  ```yaml