Update README.md

README.md

@@ -19,6 +19,8 @@ tags:
 
 # Model Details
 
+⚠️ **This is a raw, pretrained model, which should be further finetuned for most usecases.** 
+
 ## Model Description
 
 - **Developed by:** [https://www.tii.ae](https://www.tii.ae)
@@ -104,17 +106,24 @@ print(tokenizer.decode(outputs[0]))
 
 ## Training Data
 
+Falcon3-7B is trained on 15 Gigatokens of datasets comprising of web, code, STEM, high quality and mutlilingual data.
+
 ## Training Procedure
 
+Falcon3-7B is trained on 256 H100 nodes (world size 2048).
+
 ### Training Hyperparameters
 
-| **Hyperparameter** | **Value**  | **Comment**                               |
-|--------------------|------------|-------------------------------------------|
-| Precision          | `bfloat16` |                                           |
-| Optimizer          | AdamW      |                                           |
-| Max learning rate  |      | Following a WSD (warmup-stable-decay) learning rate schedule |
-| Weight decay       |        |                                           |
-| Batch size         |        |                                           |
+| **Hyperparameter** | **Value**  | **Comment**                           |
+|--------------------|------------|---------------------------------------|
+| Precision          | `bfloat16` |                                       |
+| Optimizer          | AdamW      |                                       |
+| Max learning rate  | 6e-4       | Following a WSD (warmup-stable-decay) |
+|                    |            | learning rate scheduler               |
+| Weight decay       | 1e-1       |                                       |
+| z-loss             | 1e-4       |                                       |
+| Batch size         | Variable   | Batch size was gradually increased    |
+|                    |            | during the training                   |
 
 # Evaluation