mms-1b-bemgen-combined-model

This model is a fine-tuned version of facebook/mms-1b-all on the GENBED - BEM dataset. It achieves the following results on the evaluation set:

• Loss: 0.2591
• Wer: 0.4134

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 30.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
6.7553	0.0516	100	0.8774	0.8476
0.5648	0.1031	200	0.3408	0.5032
0.4827	0.1547	300	0.3261	0.4930
0.4321	0.2063	400	0.3036	0.4854
0.4168	0.2579	500	0.2989	0.4783
0.3965	0.3094	600	0.2907	0.4513
0.4199	0.3610	700	0.2926	0.4718
0.3975	0.4126	800	0.2886	0.4459
0.3839	0.4642	900	0.2908	0.4722
0.3673	0.5157	1000	0.2836	0.4445
0.3777	0.5673	1100	0.2784	0.4365
0.3764	0.6189	1200	0.2791	0.4278
0.3918	0.6704	1300	0.2757	0.4251
0.3669	0.7220	1400	0.2721	0.4182
0.377	0.7736	1500	0.2728	0.4757
0.4174	0.8252	1600	0.2684	0.4242
0.3641	0.8767	1700	0.2649	0.4195
0.3882	0.9283	1800	0.2647	0.4125
0.3861	0.9799	1900	0.2668	0.4425
0.3647	1.0315	2000	0.2675	0.4246
0.3467	1.0830	2100	0.2629	0.4098
0.3579	1.1346	2200	0.2587	0.4186
0.3544	1.1862	2300	0.2609	0.4127
0.35	1.2378	2400	0.2592	0.4062
0.3519	1.2893	2500	0.2591	0.4135

Framework versions

• Transformers 4.48.0.dev0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0