|
--- |
|
license: mit |
|
language: |
|
- kbd |
|
datasets: |
|
- anzorq/kbd_speech |
|
- anzorq/sixuxar_yijiri_mak7 |
|
metrics: |
|
- wer |
|
pipeline_tag: automatic-speech-recognition |
|
--- |
|
# Circassian (Kabardian) ASR Model |
|
|
|
This is a fine-tuned model for Automatic Speech Recognition (ASR) in `kbd`, based on the `facebook/w2v-bert-2.0` model. |
|
|
|
The model was trained on a combination of the `anzorq/kbd_speech` (filtered on `country=russia`) and `anzorq/sixuxar_yijiri_mak7` datasets. |
|
|
|
## Model Details |
|
|
|
- **Base Model**: facebook/w2v-bert-2.0 |
|
- **Language**: Kabardian |
|
- **Task**: Automatic Speech Recognition (ASR) |
|
- **Datasets**: anzorq/kbd_speech, anzorq/sixuxar_yijiri_mak7 |
|
- **Training Steps**: 5000 |
|
|
|
## Training |
|
|
|
The model was fine-tuned using the following training arguments: |
|
|
|
```python |
|
TrainingArguments( |
|
output_dir='output', |
|
group_by_length=True, |
|
per_device_train_batch_size=8, |
|
gradient_accumulation_steps=2, |
|
evaluation_strategy="steps", |
|
num_train_epochs=10, |
|
gradient_checkpointing=True, |
|
fp16=True, |
|
save_steps=1000, |
|
eval_steps=500, |
|
logging_steps=300, |
|
learning_rate=5e-5, |
|
warmup_steps=500, |
|
save_total_limit=2, |
|
push_to_hub=True, |
|
report_to="wandb" |
|
) |
|
``` |
|
|
|
## Performance |
|
|
|
The model's performance during training: |
|
|
|
| Step | Training Loss | Validation Loss | WER | |
|
|------|---------------|-----------------|---------| |
|
| 500 | 2.859600 | inf | 0.870362| |
|
| 1000 | 0.355500 | inf | 0.703617| |
|
| 1500 | 0.247100 | inf | 0.549942| |
|
| 2000 | 0.196700 | inf | 0.471762| |
|
| 2500 | 0.181500 | inf | 0.361494| |
|
| 3000 | 0.152200 | inf | 0.314119| |
|
| 3500 | 0.135700 | inf | 0.275146| |
|
| 4000 | 0.113400 | inf | 0.252625| |
|
| 4500 | 0.102900 | inf | 0.277013| |
|
| 5000 | 0.078500 | inf | 0.250175| |
