lmeninato/t5-small-codesearchnet-python-stripped

t5-small-codesearchnet-python

This model is a fine-tuned version of t5-small on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.7802
• Bleu: 0.0027
• Rouge1: 0.2097
• Rouge2: 0.0628
• Avg Length: 15.7496

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 10
• total_train_batch_size: 80
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 15

Training results

Training Loss	Epoch	Step	Validation Loss	Bleu	Rouge1	Rouge2	Avg Length
No log	1.0	375	0.9665	0.0004	0.1696	0.0384	14.1472
2.0677	2.0	750	0.8740	0.0004	0.1758	0.0428	13.38
0.8314	3.0	1125	0.8281	0.001	0.1925	0.0505	14.8026
0.7563	4.0	1500	0.7996	0.0017	0.2033	0.0582	14.9606
0.7563	5.0	1875	0.7780	0.0022	0.2117	0.0607	14.9434
0.6959	6.0	2250	0.7587	0.002	0.2135	0.0621	14.6926
0.6591	7.0	2625	0.7545	0.002	0.2073	0.0605	15.2818
0.6205	8.0	3000	0.7472	0.0024	0.2187	0.0674	15.051
0.6205	9.0	3375	0.7506	0.0031	0.2266	0.0696	15.6286
0.5822	10.0	3750	0.7449	0.001	0.206	0.063	13.1462
0.5553	11.0	4125	0.7573	0.0027	0.2148	0.0647	16.4076
0.5295	12.0	4500	0.7677	0.0026	0.2185	0.0658	15.8986
0.5295	13.0	4875	0.7595	0.0009	0.2052	0.0618	12.3528
0.4977	14.0	5250	0.7766	0.0035	0.2177	0.0636	16.6706
0.4818	15.0	5625	0.7802	0.0027	0.2097	0.0628	15.7496

Framework versions

• Transformers 4.28.1
• Pytorch 2.0.0+cu118
• Datasets 2.12.0
• Tokenizers 0.13.3