EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions

We address text-driven expression synthesis by formulating it as a regression problem in the disentangled parameter space of a 3D Morphable Model (3DMM). To fill the gap of missing text-to-expression data, we introduce Txt2Emote, a new benchmark dataset featuring diverse 3D facial expressions paired with fine-grained textual annotations. Each expression is annotated using GPT-4o and a high-fidelity face tracker, providing two complementary forms of supervision: (1) explicit descriptions detailing facial features (e.g., mouth, nose, eyes) , and (2) implicit descriptions referencing the situational context behind the expression. Leveraging this dataset, we present EmoteGPT, a novel text-to-3D expression framework based on a Multi-Modal Large Language Model (MLLM). Our method incorporates a dedicated token to semantically ground expression representations within the MLLM, which are then decoded into 3DMM parameters via a lightweight decoder. Interestingly, we observe that extending the text-to-3DMM expression training data with large-scale image-to-3DMM data, improves the generalization ability of EmoteGPT across diverse expression descriptions. Ultimately, this enables our system to capture nuanced emotional content and context-sensitive expressions beyond the capabilities of prior CLIP-based approaches. Quantitative and qualitative evaluations demonstrate that EmoteGPT significantly outperforms state-of-the-art methods in emotion recognition and expressiveness. Integrated into avatar pipelines, our method enables photorealistic and stylized 3D avatars, as well as expressive 2D face synthesis from textual input.