Yu Guo1 Ying Shan 2 Fei Wang 1
CVPR 2023
TL;DR: single portrait image 🙎♂️ + audio 🎤 = talking head video 🎞.
-
🔥 Update LICENSE to Apache 2.0, remove non-commercial restrictation.
-
🔥 SadTalker has been officially integrated into
, where you can use sadtalker in discord server
free!!!by simple drop and also you can generate high-quality videos from text prompt. -
🔥 The extension of the stable-diffusion-webui is online. Checkout more details here.
sadtalker-webui.mp4
- 🔥
full image modeis online! checkout here for more details.
| still+enhancer in v0.0.1 | still + enhancer in v0.0.2 | input image @bagbag1815 |
|---|---|---|
still_e_n.mp4 |
full_body_2.bus_chinese_enhanced.mp4 |
 |
-
🔥 Several new mode, eg,
still mode,reference mode,resize modeare online for better and custom applications. -
🔥 Happy to see more community demos at bilibili, Youtube and twitter #sadtalker.
📋 Changelog (Previous changelog can be founded here)
-
[2023.06.12]: add more new features in WEBUI extension, see the discussion here.
-
[2023.06.05]: release a new 512 beta face model. Fixed some bugs and improve the performance.
-
[2023.04.15]: Adding automatic1111 colab by @camenduru, thanks for this awesome colab:
.
-
[2023.04.12]: adding a more detailed sd-webui installation document, fixed reinstallation problem.
-
[2023.04.12]: Fixed the sd-webui safe issues becasue of the 3rd packages, optimize the output path in
sd-webui-extension. -
[2023.04.08]: ❗️❗️❗️ In v0.0.2, we add a logo watermark to the generated video to prevent abusing since it is very realistic.
-
[2023.04.08]: v0.0.2, full image animation, adding baidu driver for download checkpoints. Optimizing the logic about enhancer.
🚧 TODO: See the Discussion OpenTalker#280
If you have any problem, please view our FAQ before opening an issue.
Tutorials from communities: 中文windows教程 | 日本語コース
-
Installing anaconda, python and git.
-
Creating the env and install the requirements.
git clone https://github.com/Winfredy/SadTalker.git
cd SadTalker
conda create -n sadtalker python=3.8
conda activate sadtalker
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113
conda install ffmpeg
pip install -r requirements.txt
### tts is optional for gradio demo.
### pip install TTS
Windows (中文windows教程):
- Install Python 3.10.6, checking "Add Python to PATH".
- Install git manually (OR
scoop install gitvia scoop). - Install
ffmpeg, following this instruction (OR usingscoop install ffmpegvia scoop). - Download our SadTalker repository, for example by running
git clone https://github.com/Winfredy/SadTalker.git. - Download the
checkpointandgfpganbelow↓. - Run
start.batfrom Windows Explorer as normal, non-administrator, user, a gradio WebUI demo will be started.
More tips about installnation on Macbook and the Docker file can be founded here
You can run the following script to put all the models in the right place.
bash scripts/download_models.shOther alternatives:
we also provide an offline patch (
gfpgan/), thus, no model will be downloaded when generating.
Google Driver: download our pre-trained model from this link (main checkpoints) and gfpgan (offline patch)
Github Release Page: download all the files from the lastest github release page, and then, put it in ./checkpoints.
百度云盘: we provided the downloaded model in checkpoints, 提取码: sadt. And gfpgan, 提取码: sadt.
Model Details
Model explains:
| Model | Description |
|---|---|
| checkpoints/mapping_00229-model.pth.tar | Pre-trained MappingNet in Sadtalker. |
| checkpoints/mapping_00109-model.pth.tar | Pre-trained MappingNet in Sadtalker. |
| checkpoints/SadTalker_V0.0.2_256.safetensors | packaged sadtalker checkpoints of old version, 256 face render). |
| checkpoints/SadTalker_V0.0.2_512.safetensors | packaged sadtalker checkpoints of old version, 512 face render). |
| gfpgan/weights | Face detection and enhanced models used in facexlib and gfpgan. |
| Model | Description |
|---|---|
| checkpoints/auido2exp_00300-model.pth | Pre-trained ExpNet in Sadtalker. |
| checkpoints/auido2pose_00140-model.pth | Pre-trained PoseVAE in Sadtalker. |
| checkpoints/mapping_00229-model.pth.tar | Pre-trained MappingNet in Sadtalker. |
| checkpoints/mapping_00109-model.pth.tar | Pre-trained MappingNet in Sadtalker. |
| checkpoints/facevid2vid_00189-model.pth.tar | Pre-trained face-vid2vid model from the reappearance of face-vid2vid. |
| checkpoints/epoch_20.pth | Pre-trained 3DMM extractor in Deep3DFaceReconstruction. |
| checkpoints/wav2lip.pth | Highly accurate lip-sync model in Wav2lip. |
| checkpoints/shape_predictor_68_face_landmarks.dat | Face landmark model used in dilb. |
| checkpoints/BFM | 3DMM library file. |
| checkpoints/hub | Face detection models used in face alignment. |
| gfpgan/weights | Face detection and enhanced models used in facexlib and gfpgan. |
The final folder will be shown as:
🔮 3. Quick Start (Best Practice).
Online: Huggingface | SDWebUI-Colab | Colab
Local Autiomatic1111 stable-diffusion webui extension: please refer to Autiomatic1111 stable-diffusion webui docs.
Local gradio demo(highly recommanded!): Similar to our hugging-face demo can be run by:
## you need manually install TTS(https://github.com/coqui-ai/TTS) via `pip install tts` in advanced.
python app_sadtalker.pyLocal gradio demo(highly recommanded!):
- windows: just double click
webui.bat, the requirements will be installed automatically. - Linux/Mac OS: run
bash webui.shto start the webui.
python inference.py --driven_audio <audio.wav> \
--source_image <video.mp4 or picture.png> \
--enhancer gfpgan The results will be saved in results/$SOME_TIMESTAMP/*.mp4.
Using --still to generate a natural full body video. You can add enhancer to improve the quality of the generated video.
python inference.py --driven_audio <audio.wav> \
--source_image <video.mp4 or picture.png> \
--result_dir <a file to store results> \
--still \
--preprocess full \
--enhancer gfpgan More examples and configuration and tips can be founded in the >>> best practice documents <<<.
If you find our work useful in your research, please consider citing:
@article{zhang2022sadtalker,
title={SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation},
author={Zhang, Wenxuan and Cun, Xiaodong and Wang, Xuan and Zhang, Yong and Shen, Xi and Guo, Yu and Shan, Ying and Wang, Fei},
journal={arXiv preprint arXiv:2211.12194},
year={2022}
}Facerender code borrows heavily from zhanglonghao's reproduction of face-vid2vid and PIRender. We thank the authors for sharing their wonderful code. In training process, We also use the model from Deep3DFaceReconstruction and Wav2lip. We thank for their wonderful work.
See also these wonderful 3rd libraries we use:
- Face Utils: https://github.com/xinntao/facexlib
- Face Enhancement: https://github.com/TencentARC/GFPGAN
- Image/Video Enhancement:https://github.com/xinntao/Real-ESRGAN
- SadTalker-Video-Lip-Sync from @Zz-ww: SadTalker for Video Lip Editing
- StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN (ECCV 2022)
- CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior (CVPR 2023)
- VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild (SIGGRAPH Asia 2022)
- DPE: Disentanglement of Pose and Expression for General Video Portrait Editing (CVPR 2023)
- 3D GAN Inversion with Facial Symmetry Prior (CVPR 2023)
- T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations (CVPR 2023)
This is not an official product of Tencent.
1. Please carefully read and comply with the open-source license applicable to this code before using it.
2. Please carefully read and comply with the intellectual property declaration applicable to this code before using it.
3. This open-source code runs completely offline and does not collect any personal information or other data. If you use this code to provide services to end-users and collect related data, please take necessary compliance measures according to applicable laws and regulations (such as publishing privacy policies, adopting necessary data security strategies, etc.). If the collected data involves personal information, user consent must be obtained (if applicable). Any legal liabilities arising from this are unrelated to Tencent.
4. Without Tencent's written permission, you are not authorized to use the names or logos legally owned by Tencent, such as "Tencent." Otherwise, you may be liable for legal responsibilities.
5. This open-source code does not have the ability to directly provide services to end-users. If you need to use this code for further model training or demos, as part of your product to provide services to end-users, or for similar use, please comply with applicable laws and regulations for your product or service. Any legal liabilities arising from this are unrelated to Tencent.
6. It is prohibited to use this open-source code for activities that harm the legitimate rights and interests of others (including but not limited to fraud, deception, infringement of others' portrait rights, reputation rights, etc.), or other behaviors that violate applicable laws and regulations or go against social ethics and good customs (including providing incorrect or false information, spreading pornographic, terrorist, and violent information, etc.). Otherwise, you may be liable for legal responsibilities.
LOGO: color and font suggestion: ChatGPT, logo font：Montserrat Alternates .
All the copyrights of the demo images and audio are from community users or the generation from stable diffusion. Feel free to contact us if you feel uncomfortable.