SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation


1 Xi'an Jiaotong University     2 Tencent AI Lab     3 Ant Group
CVPR 2023

*Indicates Equal Contribution

Abstract

Generating talking head videos through a face image and a piece of speech audio still contains many challenges. ie, unnatural head movement, distorted expression, and identity modification. We argue that these issues are mainly because of learning from the coupled 2D motion fields. On the other hand, explicitly using 3D information also suffers problems of stiff expression and incoherent video. We present SadTalker, which generates 3D motion coefficients (head pose, expression) of the 3DMM from audio and implicitly modulates a novel 3D-aware face render for talking head generation. To learn the realistic motion coefficients, we explicitly model the connections between audio and different types of motion coefficients individually. Precisely, we present ExpNet to learn the accurate facial expression from audio by distilling both coefficients and 3D-rendered faces. As for the head pose, we design PoseVAE via a conditional VAE to synthesize head motion in different styles. Finally, the generated 3D motion coefficients are mapped to the unsupervised 3D keypoints space of the proposed face render, and synthesize the final video. We conduct extensive experiments to show the superior of our method in terms of motion and video quality.

Pipeline

Video1: Talking in different languages

Video2: Speaking Chinese

Video3: Singing in different languages

Video4: Controllable eye blinking

Video5: Comparison on the HDTF dataset

Video6: Comparison on the VoxCeleb2 dataset

Video7: The results of the same driven audio with different styles

Video8: The results of different driven audios with the same style

Video9: Ablation on ExpNet

Video10: Ablation on face render

BibTeX

@misc{sadtalker,
        title={SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation},
        author={Zhang, Wenxuan and Cun, Xiaodong and Wang, Xuan and Zhang, Yong and Shen, Xi and Guo, Yu and Shan, Ying and Wang, Fei},
        year={2022},
        eprint={2211.12194},
}