Update README.md

README.md

@@ -6,4 +6,91 @@ tags:
 - jvs
 - pyopenjtalk
 - speech-to-text
----
+---
+
+# SpeechT5 (TTS task) for Japanese
+SpeechT5 model fine-tuned for speech synthesis (text-to-speech) on [JVS]("https://sites.google.com/site/shinnosuketakamichi/research-topics/jvs_corpus").
+Trained from [microsoft/speecht5_tts](https://huggingface.co/microsoft/speecht5_tts).
+Modified tokenizer powered by [Open Jtalk](https://open-jtalk.sp.nitech.ac.jp/)
+
+# Model description
+See [original model card](https://huggingface.co/microsoft/speecht5_tts#model-description)
+My modified codes licensed under MIT Licence.
+
+# Usage
+Install requirements
+```bash
+pip install transformers sentencepiece pyopnjtalk # or pyopenjtalk-prebuilt
+```
+
+Download a modified code.
+```bash
+curl -O https://huggingface.co/esnya/japanese_speecht5_tts/main/speecht5_openjtalk_tokenizer.py
+```
+
+(`SpeechToTextPipeline` is not released yet.)
+```py
+import numpy as np
+from transformers import (
+    SpeechT5ForTextToSpeech,
+    SpeechT5HifiGan,
+    SpeechT5FeatureExtractor,
+    SpeechT5Processor,
+)
+from speecht5_openjtalk_tokenizer import SpeechT5OpenjtalkTokenizer
+import soundfile
+import torch
+
+model_name = "esnya/japanese_speecht5_tts"
+with torch.no_grad():
+
+    model = SpeechT5ForTextToSpeech.from_pretrained(
+        model_name, device_map="cuda", torch_dtype=torch.bfloat16
+    )
+
+    tokenizer = SpeechT5OpenjtalkTokenizer.from_pretrained(model_name)
+    feature_extractor = SpeechT5FeatureExtractor.from_pretrained(model_name)
+    processor = SpeechT5Processor(feature_extractor, tokenizer)
+    vocoder = SpeechT5HifiGan.from_pretrained(
+        "microsoft/speecht5_hifigan", device_map="cuda", torch_dtype=torch.bfloat16
+    )
+
+    input = "吾輩は猫である。名前はまだ無い。どこで生れたかとんと見当がつかぬ。"
+    input_ids = processor(text=input, return_tensors="pt").input_ids.to(model.device)
+
+    speaker_embeddings = np.random.uniform(
+        -1, 1, (1, 16)
+    )  # (batch_size, speaker_embedding_dim = 16), first dimension means male (-1.0) / female (1.0)
+    speaker_embeddings = torch.FloatTensor(speaker_embeddings).to(
+        device=model.device, dtype=model.dtype
+    )
+
+    waveform = model.generate_speech(
+        input_ids,
+        speaker_embeddings,
+        vocoder=vocoder,
+    )
+
+    waveform = waveform / waveform.abs().max()  # normalize
+    waveform = waveform.reshape(-1).cpu().float().numpy()
+
+    soundfile.write(
+        "output.wav",
+        waveform,
+        vocoder.config.sampling_rate,
+    )
+```
+
+# Background
+
+The motivation behind developing this model stems from the noticeable lack of Japanese generation models in SpeechT5 TTS, or their scarcity at best. Additionally, the g2p functionality of Open Jtalk (pyopenjtalk) enabled us to achieve a vocabulary closely resembling English models. It's important to note that the special modifications and enhancements were primarily applied to the tokenizer. Unlike the default setup, our modified tokenizer separately extracts and retains characters other than phonation to ensure more accurate text-to-speech conversion.
+
+# Limitations
+
+One known issue with this model is that when multiple sentences are fed into it, the latter parts may result in extended silences. As a temporary solution, until this is rectified, it is recommended to split and generate each sentence individually.
+
+# License
+Model inherits [JVS Corpus](https://sites.google.com/site/shinnosuketakamichi/research-topics/jvs_corpus).
+
+# See also
+- Shinnosuke Takamichi, Kentaro Mitsui, Yuki Saito, Tomoki Koriyama, Naoko Tanji, and Hiroshi Saruwatari, "JVS corpus: free Japanese multi-speaker voice corpus," arXiv preprint, 1908.06248, Aug. 2019.