| # One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning (AAAI 2022) | |
| #### [Paper](https://arxiv.org/pdf/2112.02749.pdf) | [Demo](https://www.youtube.com/watch?v=HHj-XCXXePY) | |
| #### Requirements | |
| - Python >= 3.6 , Pytorch >= 1.8 and ffmpeg | |
| - Set up [OpenFace](https://github.com/TadasBaltrusaitis/OpenFace) | |
| - We use the OpenFace tools to extract the initial pose of the reference image | |
| - Make sure you have installed this tool, and set the `OPENFACE_POSE_EXTRACTOR_PATH` in `config.py`. For example, it should be the absolute path of the "`FeatureExtraction.exe`" for Windows. | |
| - Other requirements are listed in the 'requirements.txt' | |
| #### Pretrained Checkpoint | |
| Please download the pretrained checkpoint from [google-drive](https://drive.google.com/file/d/1mjFEozPR_2vMaVRMd9Agk_sU1VaiUYMl/view?usp=sharing) and unzip it to the directory (`/checkpoints`). Or manually modify the settings of `GENERATOR_CKPT` and `AUDIO2POSE_CKPT` in the `config.py`. | |
| #### Extract phoneme | |
| We employ the [CMU phoneset](https://github.com/cmusphinx/cmudict) to represent phonemes, the extra 'SIL' means silence. All the phonesets can be seen in '`phindex.json`'. | |
| We have extracted the phonemes for the audios in the '`sample/audio`' directory. For other audios, you can extract the phonemes by other ASR tools and then map them to the CMU phoneset. Or email to wangsuzhen@corp.netease.com for help. | |
| #### Generate Demo Results | |
| ``` | |
| python test_script.py --img_path xxx.jpg --audio_path xxx.wav --phoneme_path xxx.json --save_dir "YOUR_DIR" | |
| ``` | |
| Note that the input images must keep the same height and width and the face should be appropriately cropped as in `samples/imgs`. You can also preprocess your images with `image_preprocess.py`. | |
| #### License and Citation | |
| ``` | |
| @InProceedings{wang2021one, | |
| author = Suzhen Wang, Lincheng Li, Yu Ding, Xin Yu | |
| title = {One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning}, | |
| booktitle = {AAAI 2022}, | |
| year = {2022}, | |
| } | |
| ``` | |
| #### Acknowledgement | |
| This codebase is based on [First Order Motion Model](https://github.com/AliaksandrSiarohin/first-order-model) and [imaginaire](https://github.com/NVlabs/imaginaire), thanks for their contributions. | |