peft-starcoder-finetuned

This model is a fine-tuned version of bigcode/starcoderbase-1b on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.2169

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 50
• training_steps: 5000

Training results

Training Loss	Epoch	Step	Validation Loss
0.8313	0.9960	500	1.2189
0.5881	1.9920	1000	1.2062
0.5283	2.9880	1500	1.2020
0.4928	3.9841	2000	1.1926
0.4682	4.9801	2500	1.2326
0.4481	5.9761	3000	1.2139
0.4383	6.9721	3500	1.2099
0.4305	7.9681	4000	1.2161
0.4251	8.9641	4500	1.2138
0.423	9.9602	5000	1.2169

Framework versions

• PEFT 0.13.2
• Transformers 4.46.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.1
• Tokenizers 0.20.3