metadata

title: DetailedWER
tags:
  - evaluate
  - metric
description: >-
  Word Error Rate (WER) metric with detailed error analysis capabilities for
  speech recognition evaluation
sdk: gradio
sdk_version: 5.5.0
app_file: app.py
pinned: false

Metric Card for DetailedWER

Metric Description

DetailedWER is an enhanced version of the Word Error Rate (WER) metric used for evaluating speech recognition systems. While it calculates the standard WER score, it also provides detailed information about different types of errors (insertions, deletions, and substitutions) when requested. This makes it particularly useful for detailed analysis of speech recognition system performance.

How to Use

The metric can be loaded and used through the evaluate library:

import evaluate
wer = evaluate.load("argmaxinc/detailed-wer")
predictions = ["this is the prediction", "there is an other sample"]
references = ["this is the reference", "there is another one"]
wer_score = wer.compute(predictions=predictions, references=references)

Inputs

predictions (List[str]): List of transcriptions to score from the speech recognition system.
references (List[str]): List of reference transcriptions for each speech input.
detailed (bool, optional): Whether to return detailed error analysis. Defaults to False.

Output Values

The metric returns either a float value representing the WER score, or when detailed=True, a dictionary containing:

wer: Overall word error rate
substitution_rate: Rate of word substitutions
deletion_rate: Rate of word deletions
insertion_rate: Rate of word insertions
num_substitutions: Absolute number of substitutions
num_deletions: Absolute number of deletions
num_insertions: Absolute number of insertions
num_hits: Number of correct words

The WER score ranges from 0 to infinity, where:

0 represents perfect transcription
Lower scores are better
Scores above 1 are possible due to insertions

Values from Popular Papers

Word Error Rate is a standard metric in speech recognition. For example:

Modern speech recognition systems typically achieve WER scores between 0.02 (2%) to 0.15 (15%) on clean speech.
The exact values vary significantly based on factors like audio quality, accent, and background noise.

Examples

Basic usage:

predictions = ["this is the prediction", "there is an other sample"]
references = ["this is the reference", "there is another one"]
wer = evaluate.load("argmaxinc/detailed-wer")

# Basic WER score
wer_score = wer.compute(predictions=predictions, references=references)
# Returns: 0.5

# Detailed analysis
detailed_scores = wer.compute(predictions=predictions, references=references, detailed=True)
# Returns dictionary with detailed error analysis

Limitations and Bias

The metric treats all words equally, regardless of their importance in the sentence
It doesn't account for semantic similarity (e.g., synonyms are counted as errors)
The metric is sensitive to word order, which might not always reflect the actual quality of the transcription
Punctuation and capitalization can affect the scores if not properly normalized

Citation

@inproceedings{inproceedings,
    author = {Morris, Andrew and Maier, Viktoria and Green, Phil},
    year = {2004},
    month = {01},
    pages = {},
    title = {From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition.}
}

Further References

Word Error Rate on Wikipedia
JiWER Library - The underlying implementation used by this metric