Mistral_Sparse_refined_web_50p_cut_pre_mlp_cut_pre_attn_2024-03-24

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 2.1460

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 0
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• total_eval_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• training_steps: 5000

Training results

Training Loss	Epoch	Step	Validation Loss
2.4669	0.01	25	2.6676
2.3645	0.02	50	2.6007
2.3355	0.02	75	2.5715
2.3828	0.03	100	2.5535
2.3401	0.04	125	2.5292
2.3527	0.05	150	2.5217
2.3829	0.06	175	2.4998
2.2761	0.07	200	2.4850
2.4218	0.07	225	2.4936
2.2971	0.08	250	2.4925
2.3207	0.09	275	2.4817
2.2992	0.1	300	2.4915
2.3897	0.11	325	2.4921
2.3127	0.12	350	2.4669
2.2856	0.12	375	2.4739
2.312	0.13	400	2.4699
2.2876	0.14	425	2.4651
2.2378	0.15	450	2.4591
2.2899	0.16	475	2.4741
2.3141	0.16	500	2.4618
2.2603	0.17	525	2.4650
2.2613	0.18	550	2.4635
2.3039	0.19	575	2.4709
2.3	0.2	600	2.4532
2.2806	0.21	625	2.4611
2.3565	0.21	650	2.4614
2.2878	0.22	675	2.4600
2.2105	0.23	700	2.4468
2.3047	0.24	725	2.4557
2.2744	0.25	750	2.4510
2.327	0.26	775	2.4459
2.3467	0.26	800	2.4419
2.3345	0.27	825	2.4455
2.227	0.28	850	2.4440
2.3044	0.29	875	2.4434
2.3411	0.3	900	2.4396
2.2335	0.3	925	2.4417
2.3237	0.31	950	2.4432
2.2669	0.32	975	2.4429
2.2561	0.33	1000	2.4428
2.2862	0.34	1025	2.4387
2.1977	0.35	1050	2.4380
2.2541	0.35	1075	2.4484
2.3078	0.36	1100	2.4425
2.2566	0.37	1125	2.4418
2.3104	0.38	1150	2.4454
2.296	0.39	1175	2.4415
2.2365	0.39	1200	2.4390
2.2823	0.4	1225	2.4484
2.3187	0.41	1250	2.4303
2.2503	0.42	1275	2.4351
2.236	0.43	1300	2.4436
2.2241	0.44	1325	2.4393
2.27	0.44	1350	2.4415
2.1327	0.45	1375	2.4449
2.2509	0.46	1400	2.4427
2.3235	0.47	1425	2.4279
2.2916	0.48	1450	2.4534
2.3007	0.49	1475	2.4388
2.2441	0.49	1500	2.4388
2.2449	0.5	1525	2.4383
2.2297	0.51	1550	2.4355
2.2189	0.52	1575	2.4314
2.2334	0.53	1600	2.4335
2.3038	0.53	1625	2.4378
2.281	0.54	1650	2.4230
2.3771	0.55	1675	2.4358
2.2954	0.56	1700	2.4272
2.3176	0.57	1725	2.4333
2.2551	0.58	1750	2.4320
2.2292	0.58	1775	2.4288
2.2678	0.59	1800	2.4316
2.2064	0.6	1825	2.4344
2.285	0.61	1850	2.4272
2.264	0.62	1875	2.4307
2.1799	0.63	1900	2.4237
2.2148	0.63	1925	2.4274
2.2222	0.64	1950	2.4223
2.2573	0.65	1975	2.4314
2.2688	0.66	2000	2.4256
2.1979	0.67	2025	2.4247
2.3255	0.67	2050	2.4345
2.3069	0.68	2075	2.4306
2.2678	0.69	2100	2.4222
2.2425	0.7	2125	2.4224
2.2997	0.71	2150	2.4245
2.255	0.72	2175	2.4259
2.3064	0.72	2200	2.4281
2.2634	0.73	2225	2.4202
2.2347	0.74	2250	2.4299
2.2811	0.75	2275	2.4240
2.309	0.76	2300	2.4264
2.2937	0.77	2325	2.4218
2.244	0.77	2350	2.4227
2.2088	0.78	2375	2.4216
2.2219	0.79	2400	2.4215
2.2195	0.8	2425	2.4149
2.3011	0.81	2450	2.4246
2.2774	0.81	2475	2.4246
2.1974	0.82	2500	2.4247
2.3793	0.83	2525	2.4267
2.3	0.84	2550	2.4219
2.2795	0.85	2575	2.4232
2.2487	0.86	2600	2.4230
2.3045	0.86	2625	2.4235
2.2968	0.87	2650	2.4285
2.2446	0.88	2675	2.4235
2.3246	0.89	2700	2.4223
2.3012	0.9	2725	2.4228
2.2852	0.91	2750	2.4247
2.2467	0.91	2775	2.4261
2.2133	0.92	2800	2.4202
2.1203	0.93	2825	2.4171
2.231	0.94	2850	2.4264
2.2386	0.95	2875	2.4249
2.2277	0.95	2900	2.4227
2.2708	0.96	2925	2.4327
2.3401	0.97	2950	2.4205
2.2068	0.98	2975	2.4287
2.3009	0.99	3000	2.4215
2.2744	1.0	3025	2.4289
2.1902	1.0	3050	2.4171
2.2535	1.01	3075	2.4273
2.3347	1.02	3100	2.4219
2.2299	1.03	3125	2.4338
2.2649	1.04	3150	2.4224
2.2959	1.04	3175	2.4262
2.3125	1.05	3200	2.4176
2.29	1.06	3225	2.4178
2.2887	1.07	3250	2.4214
2.2716	1.08	3275	2.4224
2.2285	1.09	3300	2.4155
2.2141	1.09	3325	2.4250
2.2393	1.1	3350	2.4221
2.2457	1.11	3375	2.4213
2.2702	1.12	3400	2.4153
2.244	1.13	3425	2.4178
2.2556	1.14	3450	2.4241
2.2327	1.14	3475	2.4258
2.2078	1.15	3500	2.4216
2.2766	1.16	3525	2.4258
2.2011	1.17	3550	2.4166
2.2338	1.18	3575	2.4213
2.2521	1.18	3600	2.4222
2.1923	1.19	3625	2.4221
2.1908	1.2	3650	2.4229
2.2142	1.21	3675	2.4215
2.3107	1.22	3700	2.4185
2.2513	1.23	3725	2.4188
2.1988	1.23	3750	2.4244
2.3081	1.24	3775	2.4214
2.2984	1.25	3800	2.4215
2.2241	1.26	3825	2.4165
2.2694	1.27	3850	2.4204
2.268	1.28	3875	2.4217
2.2311	1.28	3900	2.4223
2.2723	1.29	3925	2.4181
2.25	1.3	3950	2.4171
2.338	1.31	3975	2.4090
2.2806	1.32	4000	2.4174
2.1563	1.32	4025	2.4264
2.2137	1.33	4050	2.4270
2.2339	1.34	4075	2.4179
2.2593	1.35	4100	2.4187
2.2901	1.36	4125	2.4308
2.3096	1.37	4150	2.4230
2.3275	1.37	4175	2.4239
2.2729	1.38	4200	2.4238
2.3258	1.39	4225	2.4158
2.2342	1.4	4250	2.4250
2.2772	1.41	4275	2.4310
2.2495	1.42	4300	2.4178
2.2578	1.42	4325	2.4200
2.245	1.43	4350	2.4237
2.2206	1.44	4375	2.4288
2.1952	1.45	4400	2.4232
2.1864	1.46	4425	2.4265
2.221	1.46	4450	2.4237
2.2828	1.47	4475	2.4329
2.2533	1.48	4500	2.4143
2.2831	1.49	4525	2.4368
2.2538	1.5	4550	2.4305
2.2023	1.51	4575	2.4267
2.2467	1.51	4600	2.4217
2.2291	1.52	4625	2.4330
2.2284	1.53	4650	2.4244
2.2123	1.54	4675	2.4322
2.3115	1.55	4700	2.4216
2.2696	1.56	4725	2.4232
2.2189	1.56	4750	2.4234
2.2323	1.57	4775	2.4265
2.279	1.58	4800	2.4213
2.2401	1.59	4825	2.4227
2.2346	1.6	4850	2.4237
2.1738	1.6	4875	2.4226
2.2086	1.61	4900	2.4137
2.2422	1.62	4925	2.4225
2.2479	1.63	4950	2.4220
2.2511	1.64	4975	2.4221
2.2086	1.65	5000	2.4272

Framework versions

• Transformers 4.36.2
• Pytorch 2.1.2+cu121
• Datasets 2.15.0
• Tokenizers 0.15.0