--- base_model: mistralai/Mixtral-8x7B-v0.1 datasets: - generator library_name: peft license: apache-2.0 tags: - trl - sft - generated_from_trainer model-index: - name: Mixtral_Alpace_v2 results: [] --- # Mixtral_Alpace_v2 This model is a fine-tuned version of [mistralai/Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) on the generator dataset. It achieves the following results on the evaluation set: - Loss: 0.3154 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2.5e-05 - train_batch_size: 8 - eval_batch_size: 8 - seed: 42 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_steps: 15 - num_epochs: 5 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 0.3573 | 0.0327 | 10 | 0.3448 | | 0.3569 | 0.0654 | 20 | 0.3446 | | 0.365 | 0.0980 | 30 | 0.3439 | | 0.341 | 0.1307 | 40 | 0.3437 | | 0.3101 | 0.1634 | 50 | 0.3428 | | 0.3538 | 0.1961 | 60 | 0.3419 | | 0.32 | 0.2288 | 70 | 0.3414 | | 0.3361 | 0.2614 | 80 | 0.3403 | | 0.3211 | 0.2941 | 90 | 0.3395 | | 0.3583 | 0.3268 | 100 | 0.3386 | | 0.3174 | 0.3595 | 110 | 0.3382 | | 0.3097 | 0.3922 | 120 | 0.3378 | | 0.33 | 0.4248 | 130 | 0.3374 | | 0.3159 | 0.4575 | 140 | 0.3368 | | 0.3636 | 0.4902 | 150 | 0.3366 | | 0.334 | 0.5229 | 160 | 0.3356 | | 0.348 | 0.5556 | 170 | 0.3353 | | 0.3296 | 0.5882 | 180 | 0.3350 | | 0.3498 | 0.6209 | 190 | 0.3338 | | 0.3461 | 0.6536 | 200 | 0.3337 | | 0.3378 | 0.6863 | 210 | 0.3335 | | 0.3114 | 0.7190 | 220 | 0.3327 | | 0.3291 | 0.7516 | 230 | 0.3324 | | 0.3189 | 0.7843 | 240 | 0.3320 | | 0.3214 | 0.8170 | 250 | 0.3311 | | 0.3117 | 0.8497 | 260 | 0.3309 | | 0.3025 | 0.8824 | 270 | 0.3310 | | 0.2679 | 0.9150 | 280 | 0.3306 | | 0.3592 | 0.9477 | 290 | 0.3304 | | 0.3097 | 0.9804 | 300 | 0.3296 | | 0.3662 | 1.0131 | 310 | 0.3295 | | 0.2969 | 1.0458 | 320 | 0.3292 | | 0.3109 | 1.0784 | 330 | 0.3290 | | 0.3369 | 1.1111 | 340 | 0.3287 | | 0.3101 | 1.1438 | 350 | 0.3287 | | 0.3264 | 1.1765 | 360 | 0.3283 | | 0.3328 | 1.2092 | 370 | 0.3278 | | 0.3234 | 1.2418 | 380 | 0.3276 | | 0.301 | 1.2745 | 390 | 0.3278 | | 0.3357 | 1.3072 | 400 | 0.3273 | | 0.3058 | 1.3399 | 410 | 0.3271 | | 0.3204 | 1.3725 | 420 | 0.3266 | | 0.3393 | 1.4052 | 430 | 0.3265 | | 0.288 | 1.4379 | 440 | 0.3265 | | 0.3121 | 1.4706 | 450 | 0.3259 | | 0.301 | 1.5033 | 460 | 0.3255 | | 0.2912 | 1.5359 | 470 | 0.3254 | | 0.3426 | 1.5686 | 480 | 0.3253 | | 0.3256 | 1.6013 | 490 | 0.3254 | | 0.291 | 1.6340 | 500 | 0.3253 | | 0.3234 | 1.6667 | 510 | 0.3249 | | 0.3024 | 1.6993 | 520 | 0.3242 | | 0.3628 | 1.7320 | 530 | 0.3240 | | 0.331 | 1.7647 | 540 | 0.3234 | | 0.321 | 1.7974 | 550 | 0.3235 | | 0.2981 | 1.8301 | 560 | 0.3230 | | 0.3369 | 1.8627 | 570 | 0.3233 | | 0.3033 | 1.8954 | 580 | 0.3227 | | 0.3578 | 1.9281 | 590 | 0.3224 | | 0.2838 | 1.9608 | 600 | 0.3224 | | 0.3026 | 1.9935 | 610 | 0.3221 | | 0.2858 | 2.0261 | 620 | 0.3228 | | 0.3001 | 2.0588 | 630 | 0.3225 | | 0.2974 | 2.0915 | 640 | 0.3219 | | 0.3071 | 2.1242 | 650 | 0.3217 | | 0.3216 | 2.1569 | 660 | 0.3217 | | 0.3056 | 2.1895 | 670 | 0.3216 | | 0.3392 | 2.2222 | 680 | 0.3215 | | 0.314 | 2.2549 | 690 | 0.3214 | | 0.3243 | 2.2876 | 700 | 0.3210 | | 0.3232 | 2.3203 | 710 | 0.3213 | | 0.3365 | 2.3529 | 720 | 0.3211 | | 0.3163 | 2.3856 | 730 | 0.3212 | | 0.3086 | 2.4183 | 740 | 0.3211 | | 0.3048 | 2.4510 | 750 | 0.3207 | | 0.299 | 2.4837 | 760 | 0.3203 | | 0.3203 | 2.5163 | 770 | 0.3203 | | 0.278 | 2.5490 | 780 | 0.3200 | | 0.3353 | 2.5817 | 790 | 0.3197 | | 0.3314 | 2.6144 | 800 | 0.3198 | | 0.2688 | 2.6471 | 810 | 0.3197 | | 0.302 | 2.6797 | 820 | 0.3194 | | 0.2843 | 2.7124 | 830 | 0.3195 | | 0.3105 | 2.7451 | 840 | 0.3190 | | 0.276 | 2.7778 | 850 | 0.3193 | | 0.3206 | 2.8105 | 860 | 0.3192 | | 0.3011 | 2.8431 | 870 | 0.3191 | | 0.3367 | 2.8758 | 880 | 0.3189 | | 0.2918 | 2.9085 | 890 | 0.3184 | | 0.3343 | 2.9412 | 900 | 0.3187 | | 0.2801 | 2.9739 | 910 | 0.3185 | | 0.2959 | 3.0065 | 920 | 0.3185 | | 0.3392 | 3.0392 | 930 | 0.3186 | | 0.3197 | 3.0719 | 940 | 0.3182 | | 0.2919 | 3.1046 | 950 | 0.3181 | | 0.3544 | 3.1373 | 960 | 0.3182 | | 0.2779 | 3.1699 | 970 | 0.3180 | | 0.3001 | 3.2026 | 980 | 0.3180 | | 0.3102 | 3.2353 | 990 | 0.3181 | | 0.3152 | 3.2680 | 1000 | 0.3182 | | 0.2962 | 3.3007 | 1010 | 0.3179 | | 0.2831 | 3.3333 | 1020 | 0.3177 | | 0.3103 | 3.3660 | 1030 | 0.3179 | | 0.2766 | 3.3987 | 1040 | 0.3175 | | 0.295 | 3.4314 | 1050 | 0.3175 | | 0.3139 | 3.4641 | 1060 | 0.3176 | | 0.299 | 3.4967 | 1070 | 0.3173 | | 0.3034 | 3.5294 | 1080 | 0.3170 | | 0.3052 | 3.5621 | 1090 | 0.3170 | | 0.2937 | 3.5948 | 1100 | 0.3170 | | 0.3046 | 3.6275 | 1110 | 0.3170 | | 0.3094 | 3.6601 | 1120 | 0.3171 | | 0.2875 | 3.6928 | 1130 | 0.3169 | | 0.2847 | 3.7255 | 1140 | 0.3169 | | 0.2947 | 3.7582 | 1150 | 0.3171 | | 0.2925 | 3.7908 | 1160 | 0.3168 | | 0.2938 | 3.8235 | 1170 | 0.3167 | | 0.2955 | 3.8562 | 1180 | 0.3167 | | 0.333 | 3.8889 | 1190 | 0.3167 | | 0.3391 | 3.9216 | 1200 | 0.3165 | | 0.2887 | 3.9542 | 1210 | 0.3166 | | 0.3067 | 3.9869 | 1220 | 0.3163 | | 0.3349 | 4.0196 | 1230 | 0.3164 | | 0.308 | 4.0523 | 1240 | 0.3162 | | 0.3252 | 4.0850 | 1250 | 0.3163 | | 0.3077 | 4.1176 | 1260 | 0.3162 | | 0.3198 | 4.1503 | 1270 | 0.3162 | | 0.2891 | 4.1830 | 1280 | 0.3162 | | 0.2712 | 4.2157 | 1290 | 0.3162 | | 0.3083 | 4.2484 | 1300 | 0.3162 | | 0.3032 | 4.2810 | 1310 | 0.3161 | | 0.3024 | 4.3137 | 1320 | 0.3159 | | 0.2966 | 4.3464 | 1330 | 0.3160 | | 0.3046 | 4.3791 | 1340 | 0.3159 | | 0.284 | 4.4118 | 1350 | 0.3158 | | 0.2885 | 4.4444 | 1360 | 0.3157 | | 0.2951 | 4.4771 | 1370 | 0.3158 | | 0.2772 | 4.5098 | 1380 | 0.3157 | | 0.305 | 4.5425 | 1390 | 0.3156 | | 0.2834 | 4.5752 | 1400 | 0.3156 | | 0.3365 | 4.6078 | 1410 | 0.3157 | | 0.3128 | 4.6405 | 1420 | 0.3158 | | 0.3004 | 4.6732 | 1430 | 0.3157 | | 0.2844 | 4.7059 | 1440 | 0.3156 | | 0.3193 | 4.7386 | 1450 | 0.3155 | | 0.3053 | 4.7712 | 1460 | 0.3156 | | 0.2961 | 4.8039 | 1470 | 0.3156 | | 0.2999 | 4.8366 | 1480 | 0.3155 | | 0.2644 | 4.8693 | 1490 | 0.3155 | | 0.311 | 4.9020 | 1500 | 0.3155 | | 0.3044 | 4.9346 | 1510 | 0.3155 | | 0.3 | 4.9673 | 1520 | 0.3156 | | 0.3378 | 5.0 | 1530 | 0.3154 | ### Framework versions - PEFT 0.12.0 - Transformers 4.44.2 - Pytorch 2.4.0+cu121 - Datasets 2.21.0 - Tokenizers 0.19.1