---
license: mit
language:
- kbd
datasets:
- anzorq/kbd_speech
- anzorq/sixuxar_yijiri_mak7
metrics:
- wer
pipeline_tag: automatic-speech-recognition
---
# Circassian (Kabardian) ASR Model

This is a fine-tuned model for Automatic Speech Recognition (ASR) in `kbd`, based on the `facebook/w2v-bert-2.0` model.

The model was trained on a combination of the `anzorq/kbd_speech` (filtered on `country=russia`) and `anzorq/sixuxar_yijiri_mak7` datasets.

## Model Details

- **Base Model**: facebook/w2v-bert-2.0
- **Language**: Kabardian
- **Task**: Automatic Speech Recognition (ASR)
- **Datasets**: anzorq/kbd_speech, anzorq/sixuxar_yijiri_mak7
- **Training Steps**: 5000

## Training

The model was fine-tuned using the following training arguments:

```python
TrainingArguments(
   output_dir='output',
   group_by_length=True,
   per_device_train_batch_size=8,
   gradient_accumulation_steps=2,
   evaluation_strategy="steps",
   num_train_epochs=10,
   gradient_checkpointing=True,
   fp16=True,
   save_steps=1000,
   eval_steps=500,
   logging_steps=300,
   learning_rate=5e-5,
   warmup_steps=500,
   save_total_limit=2,
   push_to_hub=True,
   report_to="wandb"
)
```

## Performance

The model's performance during training:

| Step | Training Loss | Validation Loss | WER     |
|------|---------------|-----------------|---------|
| 500  | 2.859600      | inf             | 0.870362|
| 1000 | 0.355500      | inf             | 0.703617|
| 1500 | 0.247100      | inf             | 0.549942|
| 2000 | 0.196700      | inf             | 0.471762|
| 2500 | 0.181500      | inf             | 0.361494|
| 3000 | 0.152200      | inf             | 0.314119|
| 3500 | 0.135700      | inf             | 0.275146|
| 4000 | 0.113400      | inf             | 0.252625|
| 4500 | 0.102900      | inf             | 0.277013|
| 5000 | 0.078500      | inf             | 0.250175|