Update README.md

README.md

@@ -20,22 +20,34 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) on the generator dataset.
 It achieves the following results on the evaluation set:
-- Loss: 2.4725
 
-## Model description
+Accuracy (Eval dataset and predict) for a sample of 10: 70.00%
 
-More information needed
+## Model description
 
-## Intended uses & limitations
+Article: https://medium.com/@frankmorales_91352/fine-tuning-meta-llama-3-8b-with-medal-a-refined-approach-for-enhanced-medical-language-b924d226b09d
 
-More information needed
 
 ## Training and evaluation data
 
-More information needed
+Article: https://medium.com/@frankmorales_91352/fine-tuning-meta-llama-3-8b-with-medal-a-refined-approach-for-enhanced-medical-language-b924d226b09d
+
+Fine-Tuning: https://github.com/frank-morales2020/MLxDL/blob/main/FineTuning_LLM_Meta_Llama_3_8B_for_MEDAL_EVALDATA.ipynb
+
+Evaluation: https://github.com/frank-morales2020/MLxDL/blob/main/Meta_Llama_3_8B_for_MEDAL_EVALUATOR_evaldata.ipynb
+
 
 ## Training procedure
 
+from transformers import EarlyStoppingCallback
+trainer.add_callback(EarlyStoppingCallback(early_stopping_patience=5))
+
+
+trainer.train()
+
+trainer.save_model()
+
+
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
@@ -50,14 +62,57 @@ The following hyperparameters were used during training:
 - lr_scheduler_warmup_ratio: 0.03
 - num_epochs: 1
 
+from transformers import TrainingArguments
+
+args = TrainingArguments(
+
+    output_dir="/content/gdrive/MyDrive/model/NEW-Meta-Llama-3-8B-MEDAL-flash-attention-2-cosine-evaldata",
+
+    #num_train_epochs=3,                     # number of training epochs
+    num_train_epochs=1,                    # number of training epochs for POC
+    per_device_train_batch_size=2,          # batch size per device during training
+    #2
+    gradient_accumulation_steps=8,          # number of steps before performing a backward/update pass
+    gradient_checkpointing=True,            # use gradient checkpointing to save memory
+    #gradient_checkpointing_kwargs={"use_reentrant": True},
+    optim="adamw_torch_fused",              # use fused adamw optimizer
+
+    #ELECTRA is trained with Adam optimizer with learning
+    #rate of 0.00002 and with batch size of 16
+
+    #trainer = Trainer(model=model, args=training_args, train_dataset=ds, optimizers=(adam_bnb_optim, None))
+    logging_steps=200,                       # log every 10 steps
+    #save_strategy="epoch",                  # save checkpoint every epoch
+
+    learning_rate=2e-4,                     # learning rate, based on QLoRA paper # i used in the first model
+    bf16=True,                              # use bfloat16 precision
+    tf32=True,                              # use tf32 precision
+    max_grad_norm=1.0,                      # max gradient norm based on QLoRA paper
+    warmup_ratio=0.05,                      # warmup ratio based on QLoRA paper = 0.03
+
+    weight_decay=0.01,
+    lr_scheduler_type="cosine",            # lr_scheduler_type="cosine" (Cosine Annealing Learning Rate)
+
+    push_to_hub=True,                       # push model to hub
+    report_to="tensorboard",                # report metrics to tensorboard
+    gradient_checkpointing_kwargs={"use_reentrant": True},
+
+    load_best_model_at_end=True,
+    logging_dir="/content/gdrive/MyDrive/model/NEW-Meta-Llama-3-8B-MEDAL-flash-attention-2-cosine-evaldata/logs",
+
+    evaluation_strategy="steps",   # Evaluate at step intervals
+    eval_steps=200,                 # Evaluate every 50 steps
+    save_strategy="steps",  # Save checkpoints at step intervals
+    save_steps=200,  # Save every 50 steps (aligned with eval_steps)
+    metric_for_best_model = "loss",
+    ]
+)  
+
 ### Training results
+Step	Training Loss	Validation Loss
+200	    2.505300	    2.382469
+3600	2.226800	    2.223289
 
-| Training Loss | Epoch  | Step | Validation Loss |
-|:-------------:|:------:|:----:|:---------------:|
-| 2.5114        | 0.0164 | 100  | 2.4076          |
-| 2.4269        | 0.0327 | 200  | 2.4570          |
-| 2.4619        | 0.0491 | 300  | 2.4668          |
-| 2.4684        | 0.0654 | 400  | 2.4725          |
 
 
 ### Framework versions