kykim0/pythia-1b-tulu-v2-mix-sys-uf-rm

b32-lr1.41e-05-s0-e2-btbinf-seed42

This model is a fine-tuned version of kykim0/pythia-1b-tulu-v2-mix on the allenai/ultrafeedback_binarized_cleaned dataset. It achieves the following results on the evaluation set:

• Loss: 0.5040
• Accuracy: 0.7458

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1.41e-05
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• total_eval_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.6627	0.0527	100	0.6306	0.6675
0.5604	0.1055	200	0.5954	0.6890
0.5743	0.1582	300	0.5773	0.6880
0.573	0.2110	400	0.5408	0.7182
0.5644	0.2637	500	0.5285	0.7361
0.5482	0.3165	600	0.5251	0.7366
0.5673	0.3692	700	0.5267	0.7279
0.5701	0.4219	800	0.5123	0.7453
0.5199	0.4747	900	0.5148	0.7376
0.5525	0.5274	1000	0.5133	0.7494
0.5197	0.5802	1100	0.5085	0.7488
0.4977	0.6329	1200	0.5146	0.7412
0.492	0.6857	1300	0.5116	0.7417
0.5046	0.7384	1400	0.5069	0.7453
0.5476	0.7911	1500	0.5044	0.7478
0.5247	0.8439	1600	0.5038	0.7468
0.5591	0.8966	1700	0.5079	0.7453
0.5228	0.9494	1800	0.5040	0.7458
0.5336	1.0021	1900	0.5045	0.7488

Framework versions

• Transformers 4.44.2
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.19.1