Update README.md

README.md

@@ -9,18 +9,18 @@ metrics:
 library_name: nemo
 pipeline_tag: automatic-speech-recognition
 ---
-#### Model
+## Model
 **Conformer-CTC** model trained on the *OOD-Speech dataset* to transcribe speech from Bangla audio. This is a large variant of the model, with ~121M parameters. To know more about the model architecture see the NeMo Documentation [here](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/models.html#conformer-ctc).
 
-#### Dataset
- The training split contains `1100+ hours` of audio data crowdsoruced from native Bangla speakers. We trained on this split for `164 epochs` , then the model was evaluated on`23+ hours` of audio across 17 diverse domains, with a validation score of `22.4% WER` .
+## Dataset
+ The training split contains `1100+ hours` of audio data crowdsoruced from native Bangla speakers. We trained on this split for `164 epochs` , then the model was evaluated on`23+ hours` of audio across 17 diverse domains .
 
-#### Usage
+## Usage
 The model can be used as a pretrained checkpoint for inference or for fine-tuning on another dataset through the [NVIDIA NeMo toolkit](https://github.com/NVIDIA/NeMo). It is recommended to install the toolkit, after installing the pyTorch package. 
 ```bash
 apt-get update && apt-get install -y libsndfile1 ffmpeg sox
 pip install Cython
-pip install nemo_toolkit['all']
+pip install nemo_toolkit['all'] #1.19.0
 pip uninstall -y torchmetrics
 pip install torchmetrics==0.9.2
 ```
@@ -29,8 +29,8 @@ After installing the required dependencies, download the .nemo file or the pretr
 import nemo.collections.asr as nemo_asr
 asr_model = nemo_asr.models.EncDecCTCModelBPE.restore_from(restore_path="<MODEL PATH>")
 ```
-##### Data Preprocessing
-Prior to feeding the input audio to the pretrained model for inference, we need to resample the audio to **16KHz**. We can achieve that using the `sox` library :
+## Data Preprocessing
+Prior to feeding the input audio to the pretrained model for training or inference, we need to resample the audio to **16KHz**. We can achieve that using the `sox` library :
 ```python
 from sox import Transformer
 if  not os.path.exists("<RESAMPLED AUDIO FILE PATH>"):
@@ -39,21 +39,60 @@ if  not os.path.exists("<RESAMPLED AUDIO FILE PATH>"):
 	tfm.channels(n_channels=1)
 	tfm.build(input_filepath= "<AUDIO FILE PATH>", output_filepath= "<RESAMPLED AUDIO FILE PATH>") 
 ```
-##### Inference using the library 
+## Training 
+We used the official [NeMo documentation on training an ASR model](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/asr/examples/kinyarwanda_asr.html) to prepare our transcript manifest and train our model. However, we did not train any custom tokenizer and instead downloaded the tokenizer from [banglaBERT-large](https://huggingface.co/csebuetnlp/banglabert_large/) for better vocabulary coverage.  For validation, we have used `29589` samples separated from the training data and processed accordingly. The final  validation score was `22.4% WER` , at epoch `164`.
+Final Training script: 
+```bash
+export TRAIN_MANIFEST_PATH="<TRAINING MANIFEST JSON>"
+export DEV_MANIFEST_PATH="<VALIDATION MANIFEST JSON>"
+export TOKENIZER_PATH="<TOKENIZER FOLDER>"
+export HYDRA_FULL_ERROR=1
+python [NEMO_GIT_FOLDER]/examples/asr/asr_ctc/speech_to_text_ctc_bpe.py --config-path=[NEMO_GIT_FOLDER]/examples/asr/conf/conformer/ --config-name=conformer_ctc_bpe \
+    model.train_ds.manifest_filepath=${TRAIN_MANIFEST_PATH} \
+    model.validation_ds.manifest_filepath=${DEV_MANIFEST_PATH} \
+    model.tokenizer.dir=${TOKENIZER_PATH} \
+    model.tokenizer.type=wpe \
+    trainer.devices=4 \
+    trainer.accelerator="gpu" \
+    trainer.strategy="ddp" \
+    trainer.max_epochs=1000 \
+    model.optim.name="adamw" \
+    model.optim.lr=0.001 \
+    model.optim.betas=[0.9,0.999] \
+    model.optim.weight_decay=0.0001 \
+    model.optim.sched.warmup_steps=2000 \
+    exp_manager.exp_dir=results/ \
+    exp_manager.create_wandb_logger=False \
+    exp_manager.resume_if_exists=true
+   ```
+## Evaluation 
+`14,016` test samples have been used to evaluate the dataset. The generated output file contains both ground truth and predicted strings. The final result is the Word Error Rate (WER) and Character Error Rate (CER) for the model. 
+```bash
+export HYDRA_FULL_ERROR=1
+python3 [NEMO_GIT_FOLDER]/examples/asr/transcribe_speech.py \
+model_path="<PRETRAINED MODEL PATH>" \
+dataset_manifest="<TEST MANIFEST JSON>" \
+output_filename=test_with_predictions.json \
+batch_size=1 
+```
+
+**Test Dataset WER/CER 69.25%/42.13%**
+## Inference
+### Inference using the library 
 ```python
 asr_model.transcribe(paths2audio_files="<LIST OF RESAMPLED AUDIO FILES>")
 ```
-##### Infer an entire batch of audio files 
+### Infer directly using the repo
 ```bash
 python [NEMO_GIT_FOLDER]/examples/asr/transcribe_speech.py \
  model_path="<PRETRAINED MODEL PATH>" \
- audio_dir="<DIRECTORY CONTAINING AUDIO FILES>" \
+ audio_dir="<DIRECTORY CONTAINING RESAMPLED AUDIO FILES>" \
  output_filename="<PATH TO OUTPUT JSON>" \
  batch_size=1
 ```
 
-#### Performance
-The model has been separately evaluated across the different domain distributions of the test data. 
+## Domain-wise Performance
+The model has also been separately evaluated across the different domain distributions of the test data. 
 For more details please refer this paper: [OOD-Speech: A Large Bengali Speech Recognition Dataset for Out-of-Distribution Benchmarking](https://arxiv.org/pdf/2305.09688.pdf) 
 ![](https://i.imgur.com/rzWzEVj.png)
 
@@ -67,4 +106,4 @@ If you this model in your work, please cite the following paper :
       archivePrefix={arXiv},
       primaryClass={eess.AS}
 }
-```
+```