Hugging Face's logo

strongpear
/
llama3.1-8B_finetune_genQA_wiki_r64_v2

PEFT
Safetensors
Generated from Trainer
Model card Files Community
llama3.1-8B_finetune_genQA_wiki_r64_v2 / README.md
strongpear's picture
strongpear
strongpear/llama3.1-8B_finetune_genQA_wiki_r64
f343e8f verified
preview code
|
raw
history blame contribute delete
20 kB
metadata
base_model: meta-llama/Llama-3.1-8B
library_name: peft
license: llama3.1
tags:
  - generated_from_trainer
model-index:
  - name: llama3.1-8B_finetune_genQA_wiki_r64_v2
    results: []

llama3.1-8B_finetune_genQA_wiki_r64_v2

This model is a fine-tuned version of meta-llama/Llama-3.1-8B on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.3744

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 3.6e-05
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 1
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
0.5552 0.0028 200 0.4789
0.3971 0.0057 400 0.4757
0.576 0.0085 600 0.4745
0.5104 0.0113 800 0.4738
0.6768 0.0142 1000 0.4725
0.4755 0.0170 1200 0.4715
0.304 0.0199 1400 0.4706
0.5975 0.0227 1600 0.4703
0.4878 0.0255 1800 0.4692
0.3835 0.0284 2000 0.4684
0.5211 0.0312 2200 0.4683
0.4565 0.0340 2400 0.4673
0.4741 0.0369 2600 0.4671
0.4993 0.0397 2800 0.4664
0.2234 0.0426 3000 0.4664
0.5535 0.0454 3200 0.4654
0.4021 0.0482 3400 0.4649
0.5119 0.0511 3600 0.4640
0.4791 0.0539 3800 0.4634
0.3927 0.0567 4000 0.4619
0.5627 0.0596 4200 0.4614
0.3563 0.0624 4400 0.4608
0.4719 0.0652 4600 0.4604
0.562 0.0681 4800 0.4593
0.5258 0.0709 5000 0.4591
0.4483 0.0738 5200 0.4583
0.4827 0.0766 5400 0.4572
0.3712 0.0794 5600 0.4577
0.3198 0.0823 5800 0.4565
0.4168 0.0851 6000 0.4567
0.4166 0.0879 6200 0.4566
0.3014 0.0908 6400 0.4557
0.397 0.0936 6600 0.4550
0.4803 0.0965 6800 0.4540
0.4119 0.0993 7000 0.4540
0.5685 0.1021 7200 0.4532
0.4883 0.1050 7400 0.4532
0.5795 0.1078 7600 0.4531
0.5066 0.1106 7800 0.4524
0.4414 0.1135 8000 0.4522
0.3382 0.1163 8200 0.4520
0.5088 0.1191 8400 0.4521
0.4083 0.1220 8600 0.4523
0.437 0.1248 8800 0.4518
0.3769 0.1277 9000 0.4513
0.3828 0.1305 9200 0.4513
0.5162 0.1333 9400 0.4506
0.3603 0.1362 9600 0.4501
0.6039 0.1390 9800 0.4500
0.3339 0.1418 10000 0.4493
0.3821 0.1447 10200 0.4491
0.478 0.1475 10400 0.4488
0.3994 0.1504 10600 0.4481
0.4471 0.1532 10800 0.4474
0.4102 0.1560 11000 0.4471
0.3164 0.1589 11200 0.4464
0.3487 0.1617 11400 0.4458
0.4359 0.1645 11600 0.4455
0.5473 0.1674 11800 0.4450
0.5629 0.1702 12000 0.4452
0.5864 0.1730 12200 0.4446
0.4339 0.1759 12400 0.4442
0.3727 0.1787 12600 0.4447
0.4229 0.1816 12800 0.4439
0.4888 0.1844 13000 0.4437
0.4252 0.1872 13200 0.4439
0.5414 0.1901 13400 0.4432
0.4577 0.1929 13600 0.4429
0.4698 0.1957 13800 0.4425
0.488 0.1986 14000 0.4423
0.3836 0.2014 14200 0.4419
0.4724 0.2043 14400 0.4415
0.4451 0.2071 14600 0.4412
0.4409 0.2099 14800 0.4408
0.4377 0.2128 15000 0.4410
0.4688 0.2156 15200 0.4408
0.4975 0.2184 15400 0.4406
0.5752 0.2213 15600 0.4405
0.4824 0.2241 15800 0.4399
0.379 0.2270 16000 0.4396
0.5237 0.2298 16200 0.4396
0.491 0.2326 16400 0.4389
0.4132 0.2355 16600 0.4388
0.4417 0.2383 16800 0.4385
0.4859 0.2411 17000 0.4380
0.4579 0.2440 17200 0.4377
0.5186 0.2468 17400 0.4366
0.4055 0.2496 17600 0.4362
0.5011 0.2525 17800 0.4359
0.3842 0.2553 18000 0.4350
0.5457 0.2582 18200 0.4352
0.5447 0.2610 18400 0.4350
0.4722 0.2638 18600 0.4349
0.4229 0.2667 18800 0.4340
0.5047 0.2695 19000 0.4338
0.2641 0.2723 19200 0.4340
0.3937 0.2752 19400 0.4333
0.4082 0.2780 19600 0.4332
0.3958 0.2809 19800 0.4317
0.4929 0.2837 20000 0.4315
0.4739 0.2865 20200 0.4313
0.5014 0.2894 20400 0.4316
0.3236 0.2922 20600 0.4308
0.3615 0.2950 20800 0.4299
0.4235 0.2979 21000 0.4297
0.2811 0.3007 21200 0.4295
0.2743 0.3035 21400 0.4287
0.4718 0.3064 21600 0.4290
0.5184 0.3092 21800 0.4289
0.3954 0.3121 22000 0.4289
0.4478 0.3149 22200 0.4288
0.4001 0.3177 22400 0.4288
0.4768 0.3206 22600 0.4288
0.3908 0.3234 22800 0.4284
0.4068 0.3262 23000 0.4280
0.3719 0.3291 23200 0.4270
0.3995 0.3319 23400 0.4266
0.4877 0.3348 23600 0.4262
0.5297 0.3376 23800 0.4262
0.4355 0.3404 24000 0.4259
0.4551 0.3433 24200 0.4255
0.4737 0.3461 24400 0.4253
0.3971 0.3489 24600 0.4252
0.3989 0.3518 24800 0.4247
0.5214 0.3546 25000 0.4246
0.4378 0.3574 25200 0.4244
0.5113 0.3603 25400 0.4246
0.4544 0.3631 25600 0.4246
0.4151 0.3660 25800 0.4244
0.3019 0.3688 26000 0.4243
0.4397 0.3716 26200 0.4242
0.5191 0.3745 26400 0.4238
0.3183 0.3773 26600 0.4237
0.203 0.3801 26800 0.4236
0.522 0.3830 27000 0.4231
0.4159 0.3858 27200 0.4232
0.348 0.3887 27400 0.4231
0.5155 0.3915 27600 0.4229
0.5407 0.3943 27800 0.4226
0.4152 0.3972 28000 0.4219
0.4148 0.4 28200 0.4213
0.5514 0.4028 28400 0.4206
0.4132 0.4057 28600 0.4201
0.4249 0.4085 28800 0.4199
0.4003 0.4113 29000 0.4199
0.3864 0.4142 29200 0.4194
0.3573 0.4170 29400 0.4192
0.4098 0.4199 29600 0.4185
0.3843 0.4227 29800 0.4187
0.5306 0.4255 30000 0.4187
0.2888 0.4284 30200 0.4183
0.2458 0.4312 30400 0.4176
0.3408 0.4340 30600 0.4169
0.4499 0.4369 30800 0.4162
0.4447 0.4397 31000 0.4157
0.523 0.4426 31200 0.4160
0.47 0.4454 31400 0.4160
0.3046 0.4482 31600 0.4155
0.4464 0.4511 31800 0.4156
0.4347 0.4539 32000 0.4152
0.5663 0.4567 32200 0.4152
0.4236 0.4596 32400 0.4150
0.4682 0.4624 32600 0.4143
0.2078 0.4652 32800 0.4138
0.4289 0.4681 33000 0.4135
0.4123 0.4709 33200 0.4134
0.2913 0.4738 33400 0.4132
0.3788 0.4766 33600 0.4131
0.3111 0.4794 33800 0.4132
0.5047 0.4823 34000 0.4124
0.3096 0.4851 34200 0.4124
0.5126 0.4879 34400 0.4121
0.5116 0.4908 34600 0.4117
0.3009 0.4936 34800 0.4115
0.5036 0.4965 35000 0.4108
0.4221 0.4993 35200 0.4109
0.5021 0.5021 35400 0.4109
0.2946 0.5050 35600 0.4108
0.4487 0.5078 35800 0.4104
0.3863 0.5106 36000 0.4103
0.3043 0.5135 36200 0.4097
0.5039 0.5163 36400 0.4092
0.537 0.5191 36600 0.4087
0.3525 0.5220 36800 0.4082
0.3099 0.5248 37000 0.4081
0.4568 0.5277 37200 0.4080
0.1907 0.5305 37400 0.4079
0.5096 0.5333 37600 0.4074
0.4411 0.5362 37800 0.4068
0.4075 0.5390 38000 0.4070
0.5252 0.5418 38200 0.4067
0.2606 0.5447 38400 0.4060
0.4556 0.5475 38600 0.4060
0.3456 0.5504 38800 0.4056
0.2922 0.5532 39000 0.4050
0.4582 0.5560 39200 0.4043
0.3284 0.5589 39400 0.4039
0.511 0.5617 39600 0.4035
0.4445 0.5645 39800 0.4035
0.2857 0.5674 40000 0.4035
0.4778 0.5702 40200 0.4030
0.4949 0.5730 40400 0.4025
0.4691 0.5759 40600 0.4021
0.6142 0.5787 40800 0.4022
0.2999 0.5816 41000 0.4011
0.4709 0.5844 41200 0.4010
0.3927 0.5872 41400 0.4004
0.4305 0.5901 41600 0.4001
0.4798 0.5929 41800 0.3988
0.5321 0.5957 42000 0.3988
0.5152 0.5986 42200 0.3987
0.3418 0.6014 42400 0.3984
0.4056 0.6043 42600 0.3984
0.2357 0.6071 42800 0.3983
0.3883 0.6099 43000 0.3980
0.4356 0.6128 43200 0.3978
0.3481 0.6156 43400 0.3977
0.3969 0.6184 43600 0.3971
0.459 0.6213 43800 0.3969
0.3185 0.6241 44000 0.3969
0.4342 0.6270 44200 0.3963
0.448 0.6298 44400 0.3961
0.4198 0.6326 44600 0.3958
0.3038 0.6355 44800 0.3957
0.3628 0.6383 45000 0.3953
0.4918 0.6411 45200 0.3954
0.5577 0.6440 45400 0.3953
0.2738 0.6468 45600 0.3952
0.481 0.6496 45800 0.3948
0.4624 0.6525 46000 0.3945
0.4184 0.6553 46200 0.3943
0.5 0.6582 46400 0.3942
0.3874 0.6610 46600 0.3941
0.4107 0.6638 46800 0.3939
0.4462 0.6667 47000 0.3938
0.4787 0.6695 47200 0.3934
0.2831 0.6723 47400 0.3932
0.3612 0.6752 47600 0.3933
0.2958 0.6780 47800 0.3928
0.2881 0.6809 48000 0.3926
0.3764 0.6837 48200 0.3927
0.3935 0.6865 48400 0.3926
0.448 0.6894 48600 0.3923
0.4448 0.6922 48800 0.3918
0.3697 0.6950 49000 0.3918
0.4227 0.6979 49200 0.3917
0.4456 0.7007 49400 0.3914
0.4413 0.7035 49600 0.3913
0.466 0.7064 49800 0.3912
0.2918 0.7092 50000 0.3910
0.3252 0.7121 50200 0.3908
0.198 0.7149 50400 0.3905
0.2525 0.7177 50600 0.3903
0.483 0.7206 50800 0.3901
0.458 0.7234 51000 0.3898
0.4184 0.7262 51200 0.3898
0.385 0.7291 51400 0.3894
0.3237 0.7319 51600 0.3893
0.3709 0.7348 51800 0.3889
0.2904 0.7376 52000 0.3890
0.362 0.7404 52200 0.3888
0.4254 0.7433 52400 0.3885
0.4099 0.7461 52600 0.3884
0.2335 0.7489 52800 0.3881
0.4477 0.7518 53000 0.3880
0.3345 0.7546 53200 0.3878
0.3674 0.7574 53400 0.3873
0.435 0.7603 53600 0.3874
0.4013 0.7631 53800 0.3873
0.3666 0.7660 54000 0.3868
0.3511 0.7688 54200 0.3864
0.4599 0.7716 54400 0.3862
0.4172 0.7745 54600 0.3859
0.3722 0.7773 54800 0.3857
0.459 0.7801 55000 0.3853
0.3358 0.7830 55200 0.3848
0.3544 0.7858 55400 0.3847
0.3254 0.7887 55600 0.3846
0.443 0.7915 55800 0.3845
0.1945 0.7943 56000 0.3844
0.4167 0.7972 56200 0.3842
0.4281 0.8 56400 0.3838
0.4496 0.8028 56600 0.3836
0.4004 0.8057 56800 0.3834
0.4789 0.8085 57000 0.3832
0.37 0.8113 57200 0.3831
0.3972 0.8142 57400 0.3830
0.4429 0.8170 57600 0.3827
0.5432 0.8199 57800 0.3825
0.4448 0.8227 58000 0.3825
0.4668 0.8255 58200 0.3820
0.3144 0.8284 58400 0.3815
0.3173 0.8312 58600 0.3812
0.4206 0.8340 58800 0.3811
0.3072 0.8369 59000 0.3810
0.3854 0.8397 59200 0.3806
0.3892 0.8426 59400 0.3805
0.4497 0.8454 59600 0.3805
0.4165 0.8482 59800 0.3805
0.3679 0.8511 60000 0.3802
0.4221 0.8539 60200 0.3796
0.4912 0.8567 60400 0.3796
0.4141 0.8596 60600 0.3794
0.3999 0.8624 60800 0.3792
0.3014 0.8652 61000 0.3788
0.5035 0.8681 61200 0.3787
0.4158 0.8709 61400 0.3784
0.3597 0.8738 61600 0.3783
0.4084 0.8766 61800 0.3782
0.3475 0.8794 62000 0.3782
0.3177 0.8823 62200 0.3781
0.3705 0.8851 62400 0.3778
0.4659 0.8879 62600 0.3776
0.4417 0.8908 62800 0.3774
0.435 0.8936 63000 0.3772
0.4289 0.8965 63200 0.3772
0.3551 0.8993 63400 0.3770
0.4331 0.9021 63600 0.3770
0.3531 0.9050 63800 0.3768
0.4663 0.9078 64000 0.3767
0.4011 0.9106 64200 0.3765
0.344 0.9135 64400 0.3763
0.3488 0.9163 64600 0.3762
0.2881 0.9191 64800 0.3761
0.4809 0.9220 65000 0.3760
0.4229 0.9248 65200 0.3759
0.4683 0.9277 65400 0.3757
0.483 0.9305 65600 0.3756
0.4342 0.9333 65800 0.3755
0.2609 0.9362 66000 0.3754
0.4405 0.9390 66200 0.3754
0.4036 0.9418 66400 0.3754
0.3688 0.9447 66600 0.3753
0.3391 0.9475 66800 0.3752
0.466 0.9504 67000 0.3751
0.4023 0.9532 67200 0.3751
0.4671 0.9560 67400 0.3750
0.2545 0.9589 67600 0.3750
0.2524 0.9617 67800 0.3749
0.3833 0.9645 68000 0.3749
0.4234 0.9674 68200 0.3749
0.4267 0.9702 68400 0.3748
0.3799 0.9730 68600 0.3747
0.2952 0.9759 68800 0.3747
0.2221 0.9787 69000 0.3746
0.4635 0.9816 69200 0.3746
0.2814 0.9844 69400 0.3745
0.3765 0.9872 69600 0.3745
0.4394 0.9901 69800 0.3745
0.4303 0.9929 70000 0.3744
0.2866 0.9957 70200 0.3744
0.4629 0.9986 70400 0.3744

Framework versions

  • PEFT 0.12.0
  • Transformers 4.45.2
  • Pytorch 2.4.0+cu121
  • Datasets 3.0.0
  • Tokenizers 0.20.1