qmeeus/whisper-small-multilingual-spoken-ner-pipeline-step-1

WhisperForSpokenNER

This model is a fine-tuned version of openai/whisper-small on the facebook/voxpopuli de+es+fr+nl dataset. It achieves the following results on the evaluation set:

• Loss: 75.5138
• F1 Score: 0.6260
• Label F1: 0.8282
• Wer: 0.1170

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 500
• training_steps: 5000
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	F1 Score	Label F1	Wer
294.2045	0.09	200	219.4521	0.3694	0.6423	0.1170
172.5491	0.18	400	158.4206	0.5112	0.7076	0.1170
152.1994	0.27	600	148.5779	0.5501	0.7391	0.1170
140.706	0.36	800	151.8108	0.5413	0.7324	0.1170
125.5897	0.45	1000	138.0534	0.5601	0.7432	0.1170
122.0436	0.54	1200	118.2416	0.5636	0.7724	0.1170
117.7194	0.63	1400	116.8705	0.5910	0.7772	0.1170
119.8977	0.71	1600	106.7047	0.5905	0.7833	0.1170
105.5846	0.8	1800	105.5354	0.5756	0.7774	0.1170
106.7833	0.89	2000	101.9971	0.5875	0.7922	0.1170
101.8875	0.98	2200	98.1714	0.5945	0.8016	0.1170
87.7438	1.07	2400	97.7943	0.6040	0.7967	0.1170
86.1916	1.16	2600	93.9310	0.6033	0.7964	0.1170
85.3271	1.25	2800	92.3677	0.6188	0.8146	0.1170
83.1457	1.34	3000	89.3458	0.6028	0.8116	0.1170
79.4126	1.43	3200	86.8935	0.6061	0.8094	0.1170
74.7596	1.52	3400	82.3525	0.6147	0.8224	0.1170
79.5526	1.61	3600	80.6440	0.6116	0.8153	0.1170
76.0212	1.7	3800	80.1555	0.6150	0.8216	0.1170
70.2905	1.79	4000	80.9369	0.6152	0.8177	0.1170
68.0936	1.88	4200	77.4738	0.6181	0.8206	0.1170
72.6116	1.97	4400	75.5524	0.6236	0.8276	0.1170
61.0175	2.06	4600	75.7015	0.6242	0.8249	0.1170
60.3508	2.14	4800	75.5521	0.6253	0.8270	0.1170
57.4103	2.23	5000	75.5138	0.6260	0.8282	0.1170

Framework versions

• Transformers 4.37.0.dev0
• Pytorch 2.1.0
• Datasets 2.14.6
• Tokenizers 0.14.1