alwaysaditi/hiporank_LED

Trying_LED_Model_Hiporank_final_setting.ipynb

This model is a fine-tuned version of allenai/led-base-16384 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 2.5873

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 5
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
2.8616	0.1008	10	2.8924
2.8923	0.2015	20	2.8183
2.9791	0.3023	30	2.7639
2.9044	0.4030	40	2.7276
2.428	0.5038	50	2.7162
2.9009	0.6045	60	2.6943
2.9211	0.7053	70	2.6682
2.7291	0.8060	80	2.6528
2.6494	0.9068	90	2.6525
2.7393	1.0076	100	2.6357
2.3916	1.1083	110	2.6384
2.4493	1.2091	120	2.6262
2.4752	1.3098	130	2.6014
2.1968	1.4106	140	2.6068
2.538	1.5113	150	2.5980
2.4522	1.6121	160	2.5959
2.4397	1.7128	170	2.6017
2.4763	1.8136	180	2.5837
1.999	1.9144	190	2.5749
2.0956	2.0151	200	2.5696
2.1285	2.1159	210	2.6099
2.1804	2.2166	220	2.5931
2.0031	2.3174	230	2.5913
2.094	2.4181	240	2.5875
2.2214	2.5189	250	2.5639
2.0745	2.6196	260	2.5723
2.3377	2.7204	270	2.5750
1.9967	2.8212	280	2.5710
2.1091	2.9219	290	2.5694
2.0384	3.0227	300	2.5606
1.9828	3.1234	310	2.5971
2.1608	3.2242	320	2.5857
1.9558	3.3249	330	2.5793
2.0719	3.4257	340	2.5769
1.8055	3.5264	350	2.5804
2.0445	3.6272	360	2.5758
2.0795	3.7280	370	2.5924
2.073	3.8287	380	2.5745
2.0314	3.9295	390	2.5697
2.0928	4.0302	400	2.5731
1.9158	4.1310	410	2.5942
2.054	4.2317	420	2.5846
1.8497	4.3325	430	2.5963
1.8353	4.4332	440	2.5943
1.9786	4.5340	450	2.5891
1.9003	4.6348	460	2.5914
1.9248	4.7355	470	2.5876
2.1843	4.8363	480	2.5873
1.9193	4.9370	490	2.5873

Framework versions

• Transformers 4.41.2
• Pytorch 2.3.0+cu121
• Datasets 2.19.2
• Tokenizers 0.19.1