README.md

---
title: DetailedWER
tags:
- evaluate
- metric
description: >-
  Word Error Rate (WER) metric with detailed error analysis capabilities for
  speech recognition evaluation
sdk: gradio
sdk_version: 5.5.0
app_file: app.py
pinned: false
---

# Metric Card for DetailedWER

## Metric Description
DetailedWER is an enhanced version of the Word Error Rate (WER) metric used for evaluating speech recognition systems. While it calculates the standard WER score, it also provides detailed information about different types of errors (insertions, deletions, and substitutions) when requested. This makes it particularly useful for detailed analysis of speech recognition system performance.

## How to Use
The metric can be loaded and used through the `evaluate` library:

```python
import evaluate
wer = evaluate.load("argmaxinc/detailed-wer")
predictions = ["this is the prediction", "there is an other sample"]
references = ["this is the reference", "there is another one"]
wer_score = wer.compute(predictions=predictions, references=references)
```

### Inputs
- **predictions** *(List[str])*: List of transcriptions to score from the speech recognition system.
- **references** *(List[str])*: List of reference transcriptions for each speech input.
- **detailed** *(bool, optional)*: Whether to return detailed error analysis. Defaults to False.

### Output Values
The metric returns either a float value representing the WER score, or when `detailed=True`, a dictionary containing:
- `wer`: Overall word error rate
- `substitution_rate`: Rate of word substitutions
- `deletion_rate`: Rate of word deletions
- `insertion_rate`: Rate of word insertions
- `num_substitutions`: Absolute number of substitutions
- `num_deletions`: Absolute number of deletions
- `num_insertions`: Absolute number of insertions
- `num_hits`: Number of correct words

The WER score ranges from 0 to infinity, where:
- 0 represents perfect transcription
- Lower scores are better
- Scores above 1 are possible due to insertions

#### Values from Popular Papers
Word Error Rate is a standard metric in speech recognition. For example:
- Modern speech recognition systems typically achieve WER scores between 0.02 (2%) to 0.15 (15%) on clean speech.
- The exact values vary significantly based on factors like audio quality, accent, and background noise.

### Examples
Basic usage:
```python
predictions = ["this is the prediction", "there is an other sample"]
references = ["this is the reference", "there is another one"]
wer = evaluate.load("argmaxinc/detailed-wer")

# Basic WER score
wer_score = wer.compute(predictions=predictions, references=references)
# Returns: 0.5

# Detailed analysis
detailed_scores = wer.compute(predictions=predictions, references=references, detailed=True)
# Returns dictionary with detailed error analysis
```

## Limitations and Bias
- The metric treats all words equally, regardless of their importance in the sentence
- It doesn't account for semantic similarity (e.g., synonyms are counted as errors)
- The metric is sensitive to word order, which might not always reflect the actual quality of the transcription
- Punctuation and capitalization can affect the scores if not properly normalized

## Citation
```bibtex
@inproceedings{inproceedings,
    author = {Morris, Andrew and Maier, Viktoria and Green, Phil},
    year = {2004},
    month = {01},
    pages = {},
    title = {From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition.}
}
```

## Further References
- [Word Error Rate on Wikipedia](https://en.wikipedia.org/wiki/Word_error_rate)
- [JiWER Library](https://github.com/jitsi/jiwer/) - The underlying implementation used by this metric