---
license: apache-2.0
datasets:
- google/cvss
language:
- en
- fr
metrics:
- bleu
---
# NAST-S2X: A Fast and End-to-End Simultaneous Speech-to-Any Translation Model
## Features
* 🤖 **An end-to-end model without intermediate text decoding**
* đź’Ş **Supports offline and streaming decoding of all modalities**
* ⚡️ **28× faster inference compared to autoregressive models**
## Examples
#### We present an example of French-to-English translation using chunk sizes of 320 ms, 2560 ms, and in offline conditions.
* Generation with chunk sizes of 320 ms and 2560 ms starts generating English translation before the source speech is complete.
* In the examples of simultaneous interpretation, the left audio channel is the input streaming speech, and the right audio channel is the simultaneous translation.
> [!NOTE]
> For a better experience, please wear headphones.
|Chunk Size 320ms | Chunk Size 2560ms | Offline|
:-------------------------:|:-------------------------: |:-------------------------:
| |
Source Speech Transcript | Reference Text Translation
:-------------------------:|:-------------------------:
Avant la fusion des communes, Rouge-Thier faisait partie de la commune de Louveigné.| before the fusion of the towns rouge thier was a part of the town of louveigne
> [!NOTE]
> For more examples, please check https://nast-s2x.github.io/.
## Performance
* ⚡️ **Lightning Fast**: 28× faster inference and competitive quality in offline speech-to-speech translation
* 👩‍💼 **Simultaneous**: Achieves high-quality simultaneous interpretation within a delay of less than 3 seconds
* 🤖 **Unified Framework**: Support end-to-end text & speech generation in one model
**Check Details** 👇
| Offline-S2S |
:-------------------------:
|
| Simul-S2S | Simul-S2T|
:-------------------------:|:-------------------------:
 | 
## Architecture
* **Fully Non-autoregressive:** Trained with **CTC-based non-monotonic latent alignment loss [(Shao and Feng, 2022)](https://arxiv.org/abs/2210.03953)** and **glancing mechanism [(Qian et al., 2021)](https://arxiv.org/abs/2008.07905)**.
* **Minimum Human Design:** Seamlessly switch between offline translation and simultaneous interpretation **by adjusting the chunk size**.
* **End-to-End:** Generate target speech **without** target text decoding.
# Sources and Usage
## Model
> [!NOTE]
> We release French-to-English speech-to-speech translation models trained on the CVSS-C dataset to reproduce results in our paper. You can train models in your desired languages by following the instructions provided below.
[🤗 Model card](https://huggingface.co/ICTNLP/NAST-S2X)
| Chunk Size | checkpoint | ASR-BLEU | ASR-BLEU (Silence Removed) | Average Lagging |
| ----------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- |---------------------------------------------------------------- |
| 320ms | [checkpoint](https://huggingface.co/ICTNLP/NAST-S2X/blob/main/chunk_320ms.pt) | 19.67 | 24.90 | -393ms |
| 1280ms | [checkpoint](https://huggingface.co/ICTNLP/NAST-S2X/blob/main/chunk_1280ms.pt) | 20.20 | 25.71 | 3330ms |
| 2560ms | [checkpoint](https://huggingface.co/ICTNLP/NAST-S2X/blob/main/chunk_2560ms.pt) | 24.88 | 26.14 | 4976ms |
| Offline | [checkpoint](https://huggingface.co/ICTNLP/NAST-S2X/blob/main/Offline.pt) | 25.82 | - | - |
| Vocoder |
| --- |
| [checkpoint](https://huggingface.co/ICTNLP/NAST-S2X/tree/main/vocoder)
|
## Inference
> [!WARNING]
> Before executing all the provided shell scripts, please ensure to replace the variables in the file with the paths specific to your machine.
### Offline Inference
* **Data preprocessing**: Follow the instructions in the [document](https://github.com/ictnlp/NAST-S2x/blob/main/Preprocessing.md).
* **Generate Acoustic Unit**: Excute [``offline_s2u_infer.sh``](https://github.com/ictnlp/NAST-S2x/blob/main/test_scripts/offline_s2u_infer.sh)
* **Generate Waveform**: Excute [``offline_wav_infer.sh``](https://github.com/ictnlp/NAST-S2x/blob/main/test_scripts/offline_wav_infer.sh)
* **Evaluation**: Using Fairseq's [ASR-BLEU evaluation toolkit](https://github.com/facebookresearch/fairseq/tree/main/examples/speech_to_speech/asr_bleu)
### Simultaneous Inference
* We use our customized fork of [``SimulEval: b43a7c``](https://github.com/Paulmzr/SimulEval/tree/b43a7c7a9f20bb4c2ff48cf1bc573b4752d7081e) to evaluate the model in simultaneous inference. This repository is built upon the official [``SimulEval: a1435b``](https://github.com/facebookresearch/SimulEval/tree/a1435b65331cac9d62ea8047fe3344153d7e7dac) and includes additional latency scorers.
* **Data preprocessing**: Follow the instructions in the [document](https://github.com/ictnlp/NAST-S2x/blob/main/Preprocessing.md).
* **Streaming Generation and Evaluation**: Excute [``streaming_infer.sh``](https://github.com/ictnlp/NAST-S2x/blob/main/test_scripts/streaming_infer.sh)
## Train your own NAST-S2X
* **Data preprocessing**: Follow the instructions in the [document](https://github.com/ictnlp/NAST-S2x/blob/main/Preprocessing.md).
* **CTC Pretraining**: Excute [``train_ctc.sh``](https://github.com/ictnlp/NAST-S2x/blob/main/train_scripts/train_ctc.sh)
* **NMLA Training**: Excute [``train_nmla.sh``](https://github.com/ictnlp/NAST-S2x/blob/main/train_scripts/train_nmla.sh)
## Citing
Please kindly cite us if you find our papers or codes useful.
```
@inproceedings{
ma2024nonautoregressive,
title={A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation},
author={Ma, Zhengrui and Fang, Qingkai and Zhang, Shaolei and Guo, Shoutao and Feng, Yang and Zhang, Min
},
booktitle={Proceedings of ACL 2024},
year={2024},
}
@inproceedings{
fang2024ctcs2ut,
title={CTC-based Non-autoregressive Textless Speech-to-Speech Translation},
author={Fang, Qingkai and Ma, Zhengrui and Zhou, Yan and Zhang, Min and Feng, Yang
},
booktitle={Findings of ACL 2024},
year={2024},
}
```