Clarify pre-tokenize before multigpu (#359)
Browse files
README.md
CHANGED
@@ -524,7 +524,14 @@ Run
|
|
524 |
accelerate launch scripts/finetune.py configs/your_config.yml
|
525 |
```
|
526 |
|
527 |
-
#### Multi-GPU
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
528 |
|
529 |
- llama FSDP
|
530 |
```yaml
|
|
|
524 |
accelerate launch scripts/finetune.py configs/your_config.yml
|
525 |
```
|
526 |
|
527 |
+
#### Multi-GPU
|
528 |
+
|
529 |
+
It is recommended to pre-tokenize dataset with the following before finetuning:
|
530 |
+
```bash
|
531 |
+
CUDA_VISIBLE_DEVICES="" accelerate ... --prepare_ds_only
|
532 |
+
```
|
533 |
+
|
534 |
+
##### Config
|
535 |
|
536 |
- llama FSDP
|
537 |
```yaml
|