Training with 90/10 Spanish dataset, 50 epochs, 3 Batch Size, reduce_lr_on_plateau

README.md

@@ -16,7 +16,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 1.2754
+- Loss: 1.8121
 
 ## Model description
 
@@ -36,23 +36,68 @@ More information needed
 
 The following hyperparameters were used during training:
 - learning_rate: 0.0001
-- train_batch_size: 2
-- eval_batch_size: 2
+- train_batch_size: 3
+- eval_batch_size: 3
 - seed: 42
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: reduce_lr_on_plateau
-- num_epochs: 5
+- num_epochs: 50
 - mixed_precision_training: Native AMP
 
 ### Training results
 
-| Training Loss | Epoch  | Step | Validation Loss |
-|:-------------:|:------:|:----:|:---------------:|
-| 1.3053        | 0.9995 | 1847 | 0.9206          |
-| 0.7158        | 1.9989 | 3694 | 0.8873          |
-| 0.3506        | 2.9984 | 5541 | 0.9619          |
-| 0.2142        | 3.9978 | 7388 | 1.1203          |
-| 0.3116        | 4.9973 | 9235 | 1.2754          |
+| Training Loss | Epoch   | Step  | Validation Loss |
+|:-------------:|:-------:|:-----:|:---------------:|
+| 0.8511        | 0.9992  | 1231  | 0.9063          |
+| 0.836         | 1.9984  | 2462  | 0.8752          |
+| 0.4611        | 2.9976  | 3693  | 0.9398          |
+| 0.3698        | 3.9968  | 4924  | 1.0870          |
+| 0.2231        | 4.9959  | 6155  | 1.1993          |
+| 0.2689        | 5.9951  | 7386  | 1.2892          |
+| 0.1975        | 6.9943  | 8617  | 1.3711          |
+| 0.1591        | 7.9935  | 9848  | 1.4205          |
+| 0.1284        | 8.9927  | 11079 | 1.4248          |
+| 0.1301        | 9.9919  | 12310 | 1.4770          |
+| 0.1298        | 10.9911 | 13541 | 1.5282          |
+| 0.1526        | 11.9903 | 14772 | 1.5175          |
+| 0.1313        | 12.9894 | 16003 | 1.5433          |
+| 0.0817        | 13.9886 | 17234 | 1.5538          |
+| 0.0869        | 14.9878 | 18465 | 1.6030          |
+| 0.0788        | 15.9870 | 19696 | 1.6366          |
+| 0.0783        | 16.9862 | 20927 | 1.6603          |
+| 0.0768        | 17.9854 | 22158 | 1.6776          |
+| 0.0751        | 18.9846 | 23389 | 1.6930          |
+| 0.0727        | 19.9838 | 24620 | 1.7093          |
+| 0.0769        | 20.9830 | 25851 | 1.7225          |
+| 0.0692        | 21.9821 | 27082 | 1.7329          |
+| 0.0713        | 22.9813 | 28313 | 1.7415          |
+| 0.0737        | 23.9805 | 29544 | 1.7592          |
+| 0.0628        | 24.9797 | 30775 | 1.7804          |
+| 0.0688        | 25.9789 | 32006 | 1.7868          |
+| 0.0648        | 26.9781 | 33237 | 1.7914          |
+| 0.0622        | 27.9773 | 34468 | 1.7932          |
+| 0.0608        | 28.9765 | 35699 | 1.7962          |
+| 0.0595        | 29.9756 | 36930 | 1.7978          |
+| 0.0575        | 30.9748 | 38161 | 1.8001          |
+| 0.1117        | 31.9740 | 39392 | 1.8011          |
+| 0.1038        | 32.9732 | 40623 | 1.8036          |
+| 0.0987        | 33.9724 | 41854 | 1.8038          |
+| 0.0922        | 34.9716 | 43085 | 1.8065          |
+| 0.0912        | 35.9708 | 44316 | 1.8072          |
+| 0.0901        | 36.9700 | 45547 | 1.8079          |
+| 0.0913        | 37.9692 | 46778 | 1.8084          |
+| 0.0821        | 38.9683 | 48009 | 1.8091          |
+| 0.0868        | 39.9675 | 49240 | 1.8096          |
+| 0.0823        | 40.9667 | 50471 | 1.8101          |
+| 0.0806        | 41.9659 | 51702 | 1.8104          |
+| 0.0813        | 42.9651 | 52933 | 1.8108          |
+| 0.0795        | 43.9643 | 54164 | 1.8112          |
+| 0.08          | 44.9635 | 55395 | 1.8116          |
+| 0.0783        | 45.9627 | 56626 | 1.8119          |
+| 0.0767        | 46.9619 | 57857 | 1.8120          |
+| 0.0778        | 47.9610 | 59088 | 1.8120          |
+| 0.0765        | 48.9602 | 60319 | 1.8121          |
+| 0.0753        | 49.9594 | 61550 | 1.8121          |
 
 
 ### Framework versions

adapter_config.json

@@ -23,13 +23,13 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "up_proj",
-    "k_proj",
     "down_proj",
+    "o_proj",
     "q_proj",
-    "v_proj",
+    "k_proj",
+    "up_proj",
     "gate_proj",
-    "o_proj"
+    "v_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d97a1fd723fc292945f8129a4694d4a02ca7df5702811260ed2dcd90769c7652
+oid sha256:39fa37bc24aae6bc2cea83c01ce946d3015c159035840a05a32495cf80ae2601
 size 1688269144

tokenizer.json

@@ -1,11 +1,6 @@
 {
   "version": "1.0",
-  "truncation": {
-    "direction": "Right",
-    "max_length": 2048,
-    "strategy": "LongestFirst",
-    "stride": 0
-  },
+  "truncation": null,
   "padding": null,
   "added_tokens": [
     {

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a251263e185cd54161c30d98501e6ab7594498a4add3666d009a3c74f91fa5c9
+oid sha256:8776e9fc710170110f17a4d462beed2ade5aa08ece691698d6f23b874d18c75e
 size 5048