Hugging Face's logo

ghost613
/
llama8_on_korean_summary

PEFT
Safetensors
Generated from Trainer
Model card Files Community
ghost613 commited on
Commit
4882582
•
1 Parent(s): eefd319

llama8_on_korean_summary

Browse files
Files changed (4) hide show
  1. README.md +40 -27
  2. adapter_config.json +3 -3
  3. adapter_model.safetensors +1 -1
  4. training_args.bin +1 -1
README.md CHANGED
@@ -16,7 +16,7 @@ should probably proofread and complete it, then remove this comment. -->
16
 
17
  This model is a fine-tuned version of [beomi/Llama-3-Open-Ko-8B-Instruct-preview](https://huggingface.co/beomi/Llama-3-Open-Ko-8B-Instruct-preview) on the None dataset.
18
  It achieves the following results on the evaluation set:
19
- - Loss: 0.8512
20
 
21
  ## Model description
22
 
@@ -44,38 +44,51 @@ The following hyperparameters were used during training:
44
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
45
  - lr_scheduler_type: linear
46
  - lr_scheduler_warmup_steps: 50
47
- - training_steps: 500
48
  - mixed_precision_training: Native AMP
49
 
50
  ### Training results
51
 
52
  | Training Loss | Epoch | Step | Validation Loss |
53
  |:-------------:|:-----:|:----:|:---------------:|
54
- | 1.5836 | 0.26 | 20 | 1.4113 |
55
- | 1.2717 | 0.53 | 40 | 1.1577 |
56
- | 1.1128 | 0.79 | 60 | 1.0624 |
57
- | 1.0331 | 1.05 | 80 | 1.0005 |
58
- | 0.9565 | 1.32 | 100 | 0.9578 |
59
- | 0.9084 | 1.58 | 120 | 0.9194 |
60
- | 0.9038 | 1.84 | 140 | 0.8943 |
61
- | 0.8157 | 2.11 | 160 | 0.8798 |
62
- | 0.7401 | 2.37 | 180 | 0.8598 |
63
- | 0.7218 | 2.63 | 200 | 0.8412 |
64
- | 0.725 | 2.89 | 220 | 0.8264 |
65
- | 0.6651 | 3.16 | 240 | 0.8289 |
66
- | 0.5981 | 3.42 | 260 | 0.8238 |
67
- | 0.5987 | 3.68 | 280 | 0.8149 |
68
- | 0.5998 | 3.95 | 300 | 0.8093 |
69
- | 0.5423 | 4.21 | 320 | 0.8248 |
70
- | 0.5188 | 4.47 | 340 | 0.8221 |
71
- | 0.5258 | 4.74 | 360 | 0.8201 |
72
- | 0.5028 | 5.0 | 380 | 0.8199 |
73
- | 0.4567 | 5.26 | 400 | 0.8421 |
74
- | 0.4558 | 5.53 | 420 | 0.8427 |
75
- | 0.4424 | 5.79 | 440 | 0.8395 |
76
- | 0.4456 | 6.05 | 460 | 0.8387 |
77
- | 0.4147 | 6.32 | 480 | 0.8517 |
78
- | 0.4254 | 6.58 | 500 | 0.8512 |
 
 
 
 
 
 
 
 
 
 
 
 
 
79
 
80
 
81
  ### Framework versions
 
16
 
17
  This model is a fine-tuned version of [beomi/Llama-3-Open-Ko-8B-Instruct-preview](https://huggingface.co/beomi/Llama-3-Open-Ko-8B-Instruct-preview) on the None dataset.
18
  It achieves the following results on the evaluation set:
19
+ - Loss: 0.8536
20
 
21
  ## Model description
22
 
 
44
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
45
  - lr_scheduler_type: linear
46
  - lr_scheduler_warmup_steps: 50
47
+ - training_steps: 760
48
  - mixed_precision_training: Native AMP
49
 
50
  ### Training results
51
 
52
  | Training Loss | Epoch | Step | Validation Loss |
53
  |:-------------:|:-----:|:----:|:---------------:|
54
+ | 1.7767 | 0.26 | 20 | 1.5096 |
55
+ | 1.2382 | 0.53 | 40 | 1.0459 |
56
+ | 0.9915 | 0.79 | 60 | 0.9451 |
57
+ | 0.9126 | 1.05 | 80 | 0.8893 |
58
+ | 0.8501 | 1.32 | 100 | 0.8515 |
59
+ | 0.8113 | 1.58 | 120 | 0.8192 |
60
+ | 0.8019 | 1.84 | 140 | 0.7939 |
61
+ | 0.7239 | 2.11 | 160 | 0.7795 |
62
+ | 0.6621 | 2.37 | 180 | 0.7594 |
63
+ | 0.6457 | 2.63 | 200 | 0.7433 |
64
+ | 0.6417 | 2.89 | 220 | 0.7281 |
65
+ | 0.5929 | 3.16 | 240 | 0.7305 |
66
+ | 0.5245 | 3.42 | 260 | 0.7242 |
67
+ | 0.5291 | 3.68 | 280 | 0.7154 |
68
+ | 0.528 | 3.95 | 300 | 0.7109 |
69
+ | 0.4696 | 4.21 | 320 | 0.7257 |
70
+ | 0.4474 | 4.47 | 340 | 0.7251 |
71
+ | 0.4572 | 4.74 | 360 | 0.7252 |
72
+ | 0.4391 | 5.0 | 380 | 0.7202 |
73
+ | 0.3794 | 5.26 | 400 | 0.7462 |
74
+ | 0.3771 | 5.53 | 420 | 0.7568 |
75
+ | 0.3754 | 5.79 | 440 | 0.7453 |
76
+ | 0.3739 | 6.05 | 460 | 0.7597 |
77
+ | 0.3179 | 6.32 | 480 | 0.7803 |
78
+ | 0.3328 | 6.58 | 500 | 0.7699 |
79
+ | 0.3259 | 6.84 | 520 | 0.7710 |
80
+ | 0.3014 | 7.11 | 540 | 0.8083 |
81
+ | 0.2759 | 7.37 | 560 | 0.8017 |
82
+ | 0.2758 | 7.63 | 580 | 0.7954 |
83
+ | 0.2798 | 7.89 | 600 | 0.8003 |
84
+ | 0.2545 | 8.16 | 620 | 0.8325 |
85
+ | 0.2451 | 8.42 | 640 | 0.8282 |
86
+ | 0.2355 | 8.68 | 660 | 0.8318 |
87
+ | 0.2382 | 8.95 | 680 | 0.8300 |
88
+ | 0.2256 | 9.21 | 700 | 0.8544 |
89
+ | 0.212 | 9.47 | 720 | 0.8532 |
90
+ | 0.2108 | 9.74 | 740 | 0.8529 |
91
+ | 0.2125 | 10.0 | 760 | 0.8536 |
92
 
93
 
94
  ### Framework versions
adapter_config.json CHANGED
@@ -19,13 +19,13 @@
19
  "rank_pattern": {},
20
  "revision": null,
21
  "target_modules": [
22
- "q_proj",
23
  "k_proj",
24
  "v_proj",
25
  "o_proj",
 
26
  "down_proj",
27
- "gate_proj",
28
- "up_proj"
29
  ],
30
  "task_type": "CAUSAL_LM",
31
  "use_rslora": false
 
19
  "rank_pattern": {},
20
  "revision": null,
21
  "target_modules": [
 
22
  "k_proj",
23
  "v_proj",
24
  "o_proj",
25
+ "up_proj",
26
  "down_proj",
27
+ "q_proj",
28
+ "gate_proj"
29
  ],
30
  "task_type": "CAUSAL_LM",
31
  "use_rslora": false
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b7ed7e8cdb06379ae863a50a21c23e056cb94b8d39d51858b1a4a4dd0bf4d336
3
  size 167832240
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:efec2080b516267b310ac04b4d4d9ba1c484a7bb349c69a0666e41efe751e236
3
  size 167832240
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:dcbca7d9e1b94b0bb41e4fd4f9b486e15af9ed15692105f1cb90113a4f59597a
3
  size 4920
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c7c965223e2c15d272fe8f1c76bc0d0e618f556716d66c44c128f135c2173279
3
  size 4920