w2v-bert-2.0-pt_pt_v2

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the common_voice_16_1 Portuguese subset using 1XRTX 3090. It achieves the following results on the test set:

• Wer: 0.10491320595991134
• Cer: 0.032070871626631914
• Bert Score: 0.9619712047981167
• Sentence Similarity: 0.93867844

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 10
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer	Bert Score
1.2735	1.0	678	0.2292	0.1589	0.0415	0.9498
0.1715	2.0	1356	0.1762	0.1283	0.0344	0.9599
0.1158	3.0	2034	0.1539	0.1100	0.0298	0.9646
0.0821	4.0	2712	0.1362	0.0949	0.0258	0.9703
0.0605	5.0	3390	0.1349	0.0860	0.0236	0.9728
0.0475	6.0	4068	0.1395	0.0871	0.0239	0.9728
0.0355	7.0	4746	0.1487	0.0837	0.0230	0.9739
0.0309	8.0	5424	0.1452	0.0873	0.0240	0.9728
0.0308	9.0	6102	0.1390	0.0843	0.0228	0.9735
0.0239	10.0	6780	0.1282	0.0832	0.0224	0.9739

Evaluation results

Test Wer	Test Cer	Test Bert Score	Runtime	Samples per second
0.09146400542583083	0.02643665913309742	0.9702128323433327	266.8185	35.282

Framework versions

• Transformers 4.38.2
• Pytorch 2.2.0
• Datasets 2.18.0
• Tokenizers 0.15.2