# F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching [![python](https://img.shields.io/badge/Python-3.10-brightgreen)](https://github.com/SWivid/F5-TTS) [![arXiv](https://img.shields.io/badge/arXiv-2410.06885-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2410.06885) [![demo](https://img.shields.io/badge/GitHub-Demo%20page-orange.svg)](https://swivid.github.io/F5-TTS/) [![hfspace](https://img.shields.io/badge/🤗-Space%20demo-yellow)](https://huggingface.co/spaces/mrfakename/E2-F5-TTS) [![msspace](https://img.shields.io/badge/🤖-Space%20demo-blue)](https://modelscope.cn/studios/modelscope/E2-F5-TTS) [![lab](https://img.shields.io/badge/X--LANCE-Lab-grey?labelColor=lightgrey)](https://x-lance.sjtu.edu.cn/) Watermark **F5-TTS**: Diffusion Transformer with ConvNeXt V2, faster trained and inference. **E2 TTS**: Flat-UNet Transformer, closest reproduction from [paper](https://arxiv.org/abs/2406.18009). **Sway Sampling**: Inference-time flow step sampling strategy, greatly improves performance ### Thanks to all the contributors ! ## News - **2024/10/08**: F5-TTS & E2 TTS base models on [🤗 Hugging Face](https://huggingface.co/SWivid/F5-TTS), [🤖 Model Scope](https://www.modelscope.cn/models/SWivid/F5-TTS_Emilia-ZH-EN), [🟣 Wisemodel](https://wisemodel.cn/models/SJTU_X-LANCE/F5-TTS_Emilia-ZH-EN). ## Installation ```bash # Create a python 3.10 conda env (you could also use virtualenv) conda create -n f5-tts python=3.10 conda activate f5-tts # Install pytorch with your CUDA version, e.g. pip install torch==2.3.0+cu118 torchaudio==2.3.0+cu118 --extra-index-url https://download.pytorch.org/whl/cu118 ``` Then you can choose from a few options below: ### 1. As a pip package (if just for inference) ```bash pip install git+https://github.com/SWivid/F5-TTS.git ``` ### 2. Local editable (if also do training, finetuning) ```bash git clone https://github.com/SWivid/F5-TTS.git cd F5-TTS # git submodule update --init --recursive # (optional, if need bigvgan) pip install -e . ``` If initialize submodule, you should add the following code at the beginning of `src/third_party/BigVGAN/bigvgan.py`. ```python import os import sys sys.path.append(os.path.dirname(os.path.abspath(__file__))) ``` ### 3. Docker usage ```bash # Build from Dockerfile docker build -t f5tts:v1 . # Or pull from GitHub Container Registry docker pull ghcr.io/swivid/f5-tts:main ``` ## Inference ### 1. Gradio App Currently supported features: - Basic TTS with Chunk Inference - Multi-Style / Multi-Speaker Generation - Voice Chat powered by Qwen2.5-3B-Instruct - [Custom inference with more language support](src/f5_tts/infer/SHARED.md) ```bash # Launch a Gradio app (web interface) f5-tts_infer-gradio # Specify the port/host f5-tts_infer-gradio --port 7860 --host 0.0.0.0 # Launch a share link f5-tts_infer-gradio --share ``` ### 2. CLI Inference ```bash # Run with flags # Leave --ref_text "" will have ASR model transcribe (extra GPU memory usage) f5-tts_infer-cli \ --model "F5-TTS" \ --ref_audio "ref_audio.wav" \ --ref_text "The content, subtitle or transcription of reference audio." \ --gen_text "Some text you want TTS model generate for you." # Run with default setting. src/f5_tts/infer/examples/basic/basic.toml f5-tts_infer-cli # Or with your own .toml file f5-tts_infer-cli -c custom.toml # Multi voice. See src/f5_tts/infer/README.md f5-tts_infer-cli -c src/f5_tts/infer/examples/multi/story.toml ``` ### 3. More instructions - In order to have better generation results, take a moment to read [detailed guidance](src/f5_tts/infer). - The [Issues](https://github.com/SWivid/F5-TTS/issues?q=is%3Aissue) are very useful, please try to find the solution by properly searching the keywords of problem encountered. If no answer found, then feel free to open an issue. ## Training ### 1. Gradio App Read [training & finetuning guidance](src/f5_tts/train) for more instructions. ```bash # Quick start with Gradio web interface f5-tts_finetune-gradio ``` ## [Evaluation](src/f5_tts/eval) ## Development Use pre-commit to ensure code quality (will run linters and formatters automatically) ```bash pip install pre-commit pre-commit install ``` When making a pull request, before each commit, run: ```bash pre-commit run --all-files ``` Note: Some model components have linting exceptions for E722 to accommodate tensor notation ## Acknowledgements - [E2-TTS](https://arxiv.org/abs/2406.18009) brilliant work, simple and effective - [Emilia](https://arxiv.org/abs/2407.05361), [WenetSpeech4TTS](https://arxiv.org/abs/2406.05763) valuable datasets - [lucidrains](https://github.com/lucidrains) initial CFM structure with also [bfs18](https://github.com/bfs18) for discussion - [SD3](https://arxiv.org/abs/2403.03206) & [Hugging Face diffusers](https://github.com/huggingface/diffusers) DiT and MMDiT code structure - [torchdiffeq](https://github.com/rtqichen/torchdiffeq) as ODE solver, [Vocos](https://huggingface.co/charactr/vocos-mel-24khz) as vocoder - [FunASR](https://github.com/modelscope/FunASR), [faster-whisper](https://github.com/SYSTRAN/faster-whisper), [UniSpeech](https://github.com/microsoft/UniSpeech) for evaluation tools - [ctc-forced-aligner](https://github.com/MahmoudAshraf97/ctc-forced-aligner) for speech edit test - [mrfakename](https://x.com/realmrfakename) huggingface space demo ~ - [f5-tts-mlx](https://github.com/lucasnewman/f5-tts-mlx/tree/main) Implementation with MLX framework by [Lucas Newman](https://github.com/lucasnewman) - [F5-TTS-ONNX](https://github.com/DakeQQ/F5-TTS-ONNX) ONNX Runtime version by [DakeQQ](https://github.com/DakeQQ) ## Citation If our work and codebase is useful for you, please cite as: ``` @article{chen-etal-2024-f5tts, title={F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching}, author={Yushen Chen and Zhikang Niu and Ziyang Ma and Keqi Deng and Chunhui Wang and Jian Zhao and Kai Yu and Xie Chen}, journal={arXiv preprint arXiv:2410.06885}, year={2024}, } ``` ## License Our code is released under MIT License. The pre-trained models are licensed under the CC-BY-NC license due to the training data Emilia, which is an in-the-wild dataset. Sorry for any inconvenience this may cause.