navidved commited on
Commit
46170ca
β€’
1 Parent(s): a88287a

Update constants.py

Browse files
Files changed (1) hide show
  1. constants.py +27 -17
constants.py CHANGED
@@ -1,6 +1,6 @@
1
  from pathlib import Path
2
 
3
- # Directory where request by models are stored
4
  DIR_OUTPUT_REQUESTS = Path("requested_models")
5
  EVAL_REQUESTS_PATH = Path("eval_requests")
6
 
@@ -8,17 +8,21 @@ EVAL_REQUESTS_PATH = Path("eval_requests")
8
  # Text definitions #
9
  ##########################
10
 
11
- banner_url = "https://huggingface.co/datasets/vargha/persian_asr_leaderboard/resolve/main/banner.png"
12
  BANNER = f'<div style="display: flex; justify-content: space-around;"><img src="{banner_url}" alt="Banner" style="width: 40vw; min-width: 300px; max-width: 600px;"> </div>'
13
 
14
- INTRODUCTION_TEXT = "πŸ“ The πŸ€— Persian ASR Leaderboard ranks and evaluates speech recognition models \
15
- on the Hugging Face Hub using the Persian Common Voice dataset. \
16
- \nWe report the [WER](https://huggingface.co/spaces/evaluate-metric/wer) and [CER](https://huggingface.co/spaces/evaluate-metric/cer) metrics (⬇️ lower the better). Models are ranked based on their WER, from lowest to highest. Check the πŸ“ˆ Metrics tab to understand how the models are evaluated. \
17
- \nIf you want results for a model that is not listed here, you can submit a request for it to be included βœ‰οΈβœ¨."
 
 
 
 
18
 
19
  CITATION_TEXT = """@misc{persian-asr-leaderboard,
20
  title = {Persian Automatic Speech Recognition Leaderboard},
21
- author = {Your Name},
22
  year = 2024,
23
  publisher = {Hugging Face},
24
  howpublished = "\\url{https://huggingface.co/spaces/your-username/persian_asr_leaderboard}"
@@ -26,22 +30,28 @@ CITATION_TEXT = """@misc{persian-asr-leaderboard,
26
  """
27
 
28
  METRICS_TAB_TEXT = """
29
- # Metrics and Dataset
30
-
31
  ## Metrics
 
 
 
32
 
33
- We evaluate models using the Word Error Rate (WER) and Character Error Rate (CER) metrics. Both metrics are used to measure the accuracy of automatic speech recognition systems.
34
 
35
- - **Word Error Rate (WER)**: Calculates the percentage of words that were incorrectly predicted. A lower WER indicates better performance.
36
- - **Character Error Rate (CER)**: Similar to WER but operates at the character level, which can be more informative for languages with rich morphology like Persian.
37
 
38
- ## Dataset
 
39
 
40
- We use the [Persian Common Voice](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) dataset for evaluation. The dataset consists of diverse speech recordings from various speakers, making it a good benchmark for Persian ASR models.
 
41
 
42
- ## How to Submit Your Model
 
43
 
44
- To submit your model for evaluation, go to the "βœ‰οΈβœ¨ Request a model here!" tab and enter your model's name in the format `username/model_name`. Your model should be hosted on the Hugging Face Hub.
 
45
 
 
46
  """
47
-
 
1
  from pathlib import Path
2
 
3
+ # Directory where model requests are stored
4
  DIR_OUTPUT_REQUESTS = Path("requested_models")
5
  EVAL_REQUESTS_PATH = Path("eval_requests")
6
 
 
8
  # Text definitions #
9
  ##########################
10
 
11
+ banner_url = "https://cdn-thumbnails.huggingface.co/social-thumbnails/spaces/k2-fsa/automatic-speech-recognition.png"
12
  BANNER = f'<div style="display: flex; justify-content: space-around;"><img src="{banner_url}" alt="Banner" style="width: 40vw; min-width: 300px; max-width: 600px;"> </div>'
13
 
14
+ INTRODUCTION_TEXT = """
15
+ πŸ“ The πŸ€— **Persian Automatic Speech Recognition (ASR) Leaderboard** serves as an authoritative ranking of speech recognition models hosted on the Hugging Face Hub, evaluated using multiple Persian speech datasets.
16
+ We report two key performance metrics: [Word Error Rate (WER)](https://huggingface.co/spaces/evaluate-metric/wer) and [Character Error Rate (CER)](https://huggingface.co/spaces/evaluate-metric/cer), where lower scores indicate better performance.
17
+
18
+ The leaderboard primarily ranks models based on WER, from lowest to highest. You can refer to the πŸ“ˆ **Metrics** tab for a detailed explanation of how these models are evaluated.
19
+
20
+ If there is a model you'd like to see ranked but is not listed here, you may submit a request for evaluation by following the instructions in the "Request a Model" tab βœ‰οΈβœ¨.
21
+ """
22
 
23
  CITATION_TEXT = """@misc{persian-asr-leaderboard,
24
  title = {Persian Automatic Speech Recognition Leaderboard},
25
+ author = {Navid},
26
  year = 2024,
27
  publisher = {Hugging Face},
28
  howpublished = "\\url{https://huggingface.co/spaces/your-username/persian_asr_leaderboard}"
 
30
  """
31
 
32
  METRICS_TAB_TEXT = """
33
+ # Evaluation Metrics and Datasets
 
34
  ## Metrics
35
+ We employ the following metrics to evaluate the performance of Automatic Speech Recognition (ASR) models:
36
+ - **Word Error Rate (WER)**: WER quantifies the proportion of incorrectly predicted words in a transcription. A lower WER reflects higher accuracy.
37
+ - **Character Error Rate (CER)**: CER measures errors at the character level, providing a more granular view of transcription accuracy, especially in morphologically rich languages such as Persian.
38
 
39
+ Both metrics are widely used in ASR evaluation, offering a comprehensive view of model performance.
40
 
41
+ ## Datasets
42
+ The models on the Persian ASR Leaderboard are evaluated using a diverse range of datasets to ensure robust performance across different speech conditions:
43
 
44
+ 1. **Persian Common Voice (Mozilla)**
45
+ Available [here](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0), this dataset is part of the broader Common Voice project and features speech data from various speakers, accents, and environments. It serves as a representative benchmark for Persian ASR.
46
 
47
+ 2. **ASR Farsi YouTube Chunked 10 Seconds**
48
+ This dataset, available on Hugging Face [here](https://huggingface.co/datasets/pourmand1376/asr-farsi-youtube-chunked-10-seconds), consists of transcribed speech from Persian YouTube videos, split into 10-second segments. It introduces variability in audio quality and speaker demographics, adding to the challenge of accurate recognition.
49
 
50
+ 3. **Persian-ASR-Test-Set (Private)**
51
+ This private dataset is designed for in-depth model testing and evaluation. It contains curated, real-world Persian speech data from various contexts and speaker backgrounds. Access to this dataset is restricted, ensuring models are evaluated on a controlled, high-quality speech corpus.
52
 
53
+ ## How to Submit Your Model for Evaluation
54
+ To request that a model be included on this leaderboard, please submit its name in the following format: `username/model_name`. Models should be available on the Hugging Face Hub for public access.
55
 
56
+ Simply navigate to the "Request a Model" tab, enter the details, and your model will be evaluated at the next available opportunity.
57
  """