Update README.md
Browse files
README.md
CHANGED
|
@@ -184,7 +184,7 @@ Please refer to [togethercomputer/RedPajama-Data-1T](https://huggingface.co/data
|
|
| 184 |
- **Optimizer:** Apex FusedAdam
|
| 185 |
- **Parallelism:** Pipeline parallel 12, tensor parallel 2
|
| 186 |
- **Gradient Accumulations**: 8 (global batch size 4M tokens)
|
| 187 |
-
- **Num of Tokens:**
|
| 188 |
- **Learning rate:** 0.00012
|
| 189 |
|
| 190 |
## Benchmark
|
|
|
|
| 184 |
- **Optimizer:** Apex FusedAdam
|
| 185 |
- **Parallelism:** Pipeline parallel 12, tensor parallel 2
|
| 186 |
- **Gradient Accumulations**: 8 (global batch size 4M tokens)
|
| 187 |
+
- **Num of Tokens:** 1.001T Tokens
|
| 188 |
- **Learning rate:** 0.00012
|
| 189 |
|
| 190 |
## Benchmark
|