Google_bert-base-uncased

This model is a fine-tuned version of google-bert/bert-base-uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.0596

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: inverse_sqrt
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
2.2234	0.024	30	0.9557
0.4601	0.048	60	0.4359
0.3149	0.072	90	0.2795
0.2403	0.096	120	0.2142
0.1876	0.12	150	0.1714
0.1691	0.144	180	0.1488
0.1394	0.168	210	0.1273
0.1264	0.192	240	0.1160
0.1113	0.216	270	0.1079
0.1148	0.24	300	0.0992
0.0995	0.264	330	0.0940
0.096	0.288	360	0.0941
0.0954	0.312	390	0.0854
0.089	0.336	420	0.0899
0.0826	0.36	450	0.0841
0.0872	0.384	480	0.0811
0.0794	0.408	510	0.0759
0.0756	0.432	540	0.0766
0.0826	0.456	570	0.0729
0.0841	0.48	600	0.0715
0.076	0.504	630	0.0737
0.0746	0.528	660	0.0691
0.0719	0.552	690	0.0697
0.0722	0.576	720	0.0673
0.0713	0.6	750	0.0656
0.0671	0.624	780	0.0652
0.0741	0.648	810	0.0675
0.0723	0.672	840	0.0663
0.0687	0.696	870	0.0649
0.067	0.72	900	0.0637
0.0623	0.744	930	0.0643
0.0599	0.768	960	0.0643
0.0686	0.792	990	0.0624
0.0638	0.816	1020	0.0623
0.0565	0.84	1050	0.0626
0.0614	0.864	1080	0.0615
0.063	0.888	1110	0.0592
0.0592	0.912	1140	0.0618
0.0687	0.936	1170	0.0618
0.0577	0.96	1200	0.0600
0.0629	0.984	1230	0.0596

Framework versions

• Transformers 4.42.3
• Pytorch 2.3.1+cu118
• Datasets 2.20.0
• Tokenizers 0.19.1