patrickvonplaten commited on
Commit
0aa5934
1 Parent(s): adc5417

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +110 -0
README.md ADDED
@@ -0,0 +1,110 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ - ar
5
+ datasets:
6
+ - covost2
7
+ - librispeech_asr
8
+ tags:
9
+ - audio
10
+ - speech-translation
11
+ - automatic-speech-recognition
12
+ - speech2text2
13
+ license: MIT
14
+ pipeline_tag: automatic-speech-recognition
15
+ widget:
16
+ - label: Librispeech sample 1
17
+ src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
18
+ - label: Librispeech sample 2
19
+ src: https://cdn-media.huggingface.co/speech_samples/sample2.flac
20
+ ---
21
+
22
+
23
+ # S2T2-Wav2Vec2-CoVoST2-EN-AR-ST
24
+
25
+ `s2t-wav2vec2-large-en-ar` is a Speech to Text Transformer (S2T) model trained for end-to-end Speech Translation (ST).
26
+ The S2T model was proposed in [Large-Scale Self- and Semi-Supervised Learning for Speech Translation](https://arxiv.org/pdf/2104.06678.pdf) and officially released in
27
+ [Fairseq](https://github.com/pytorch/fairseq/blob/6f847c8654d56b4d1b1fbacec027f47419426ddb/fairseq/models/wav2vec/wav2vec2_asr.py#L266).
28
+
29
+
30
+ ## Model description
31
+
32
+ S2T2 is a transformer-based seq2seq (speech encoder-decoder) model designed for end-to-end Automatic Speech Recognition (ASR) and Speech
33
+ Translation (ST). It uses a pretrained [Wav2Vec2](https://huggingface.co/transformers/model_doc/wav2vec2.html) as the encoder and a transformer-based decoder. The model is trained with standard autoregressive cross-entropy loss and generates the translations autoregressively.
34
+
35
+ ## Intended uses & limitations
36
+
37
+ This model can be used for end-to-end English speech to German text translation.
38
+ See the [model hub](https://huggingface.co/models?filter=speech2text2) to look for other S2T2 checkpoints.
39
+
40
+
41
+ ### How to use
42
+
43
+ As this a standard sequence to sequence transformer model, you can use the `generate` method to generate the
44
+ transcripts by passing the speech features to the model.
45
+
46
+ You can use the model directly via the ASR pipeline
47
+
48
+ ```python
49
+ from datasets import load_dataset
50
+ from transformers import pipeline
51
+
52
+ librispeech_en = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
53
+ asr = pipeline("automatic-speech-recognition", model="facebook/s2t-wav2vec2-large-en-ar", feature_extractor="facebook/s2t-wav2vec2-large-en-ar")
54
+
55
+ translation_de = asr(librispeech_en[0]["file"])
56
+ ```
57
+
58
+ or step-by-step as follows:
59
+
60
+ ```python
61
+ import torch
62
+ from transformers import Speech2Text2Processor, SpeechEncoderDecoder
63
+ from datasets import load_dataset
64
+
65
+ import soundfile as sf
66
+ model = SpeechEncoderDecoder.from_pretrained("facebook/s2t-wav2vec2-large-en-ar")
67
+ processor = Speech2Text2Processor.from_pretrained("facebook/s2t-wav2vec2-large-en-ar")
68
+
69
+ def map_to_array(batch):
70
+ speech, _ = sf.read(batch["file"])
71
+ batch["speech"] = speech
72
+ return batch
73
+
74
+ ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
75
+ ds = ds.map(map_to_array)
76
+
77
+ inputs = processor(ds["speech"][0], sampling_rate=16_000, return_tensors="pt")
78
+ generated_ids = model.generate(input_ids=inputs["input_features"], attention_mask=inputs["attention_mask"])
79
+ transcription = processor.batch_decode(generated_ids)
80
+ ```
81
+
82
+ ## Evaluation results
83
+
84
+ CoVoST-V2 test results for en-ar (BLEU score): **20.8**
85
+
86
+ For more information, please have a look at the [official paper](https://arxiv.org/pdf/2104.06678.pdf)
87
+
88
+ ### BibTeX entry and citation info
89
+
90
+ ```bibtex
91
+ @article{DBLP:journals/corr/abs-2104-06678,
92
+ author = {Changhan Wang and
93
+ Anne Wu and
94
+ Juan Miguel Pino and
95
+ Alexei Baevski and
96
+ Michael Auli and
97
+ Alexis Conneau},
98
+ title = {Large-Scale Self- and Semi-Supervised Learning for Speech Translation},
99
+ journal = {CoRR},
100
+ volume = {abs/2104.06678},
101
+ year = {2021},
102
+ url = {https://arxiv.org/abs/2104.06678},
103
+ archivePrefix = {arXiv},
104
+ eprint = {2104.06678},
105
+ timestamp = {Thu, 12 Aug 2021 15:37:06 +0200},
106
+ biburl = {https://dblp.org/rec/journals/corr/abs-2104-06678.bib},
107
+ bibsource = {dblp computer science bibliography, https://dblp.org}
108
+ }
109
+
110
+ ```