psvishnu/Phi-3.5-mini-instruct-qlora

README.md

@@ -18,7 +18,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [microsoft/Phi-3.5-mini-instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.8409
+- Loss: 0.8569
 
 ## Model description
 
@@ -50,18 +50,18 @@ The following hyperparameters were used during training:
 
 | Training Loss | Epoch  | Step | Validation Loss |
 |:-------------:|:------:|:----:|:---------------:|
-| 1.56          | 0.3333 | 10   | 1.2560          |
-| 1.0643        | 0.6667 | 20   | 0.9815          |
-| 0.8887        | 1.0    | 30   | 0.9018          |
-| 0.8015        | 1.3333 | 40   | 0.8677          |
-| 0.7439        | 1.6667 | 50   | 0.8454          |
-| 0.7016        | 2.0    | 60   | 0.8353          |
-| 0.6358        | 2.3333 | 70   | 0.8367          |
-| 0.6145        | 2.6667 | 80   | 0.8457          |
-| 0.6228        | 3.0    | 90   | 0.8336          |
-| 0.5617        | 3.3333 | 100  | 0.8385          |
-| 0.5535        | 3.6667 | 110  | 0.8405          |
-| 0.5772        | 4.0    | 120  | 0.8409          |
+| 1.6558        | 0.3333 | 10   | 1.4457          |
+| 1.2244        | 0.6667 | 20   | 1.1363          |
+| 1.0198        | 1.0    | 30   | 1.0057          |
+| 0.92          | 1.3333 | 40   | 0.9328          |
+| 0.8365        | 1.6667 | 50   | 0.8991          |
+| 0.7908        | 2.0    | 60   | 0.8744          |
+| 0.7432        | 2.3333 | 70   | 0.8794          |
+| 0.7421        | 2.6667 | 80   | 0.8692          |
+| 0.739         | 3.0    | 90   | 0.8599          |
+| 0.6996        | 3.3333 | 100  | 0.8603          |
+| 0.7016        | 3.6667 | 110  | 0.8560          |
+| 0.7205        | 4.0    | 120  | 0.8569          |
 
 
 ### Framework versions

adapter_config.json

@@ -20,10 +20,8 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "o_proj",
-    "gate_up_proj",
     "qkv_proj",
-    "down_proj"
+    "o_proj"
   ],
   "task_type": "CAUSAL_LM",
   "use_dora": false,

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:633fa7c4d646f22871a1bba864a5316f94fde7e0647ddc374fa7eab63c64e050
-size 100697728
+oid sha256:2b7193a9250b868e45abb649dca070d3a2d1b5f3097562fbca78a91c4777a6f3
+size 37766064

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:187c1d080bb451f2c2338d7dc5fce9e06a0a19044b2a90391a1b4a4bedb64240
+oid sha256:e6f77cc20f3a9851b6b2714db72118558e4efb0e9002500e9a0100018c491dfd
 size 5496