TEMOS: Generating diverse human motions from textual descriptions

By Mathis Petrovich, Michael J. Black, Gul Varol. European Conference on Computer Vision, 2022

627 citations recorded by the source corpus.

We address the problem of generating diverse 3D human motions from textual descriptions. This challenging task requires joint modeling of both modalities: understanding and extracting useful human-centric information from the text, and then generating plausible and realistic sequences of human poses. In contrast to most previous work which focuses on generating a single, deterministic, motion from a textual description, we design a variational approach that can produce multiple diverse human motions. We propose TEMOS, a text-conditioned generative model leveraging variational autoencoder (VAE) training with human motion data, in combination with a text encoder that produces distribution parameters compatible with the VAE latent space. We show the TEMOS framework can produce both skeleton-based animations as in prior work, as well more expressive SMPL body motions. We evaluate our approach on the KIT Motion-Language benchmark and, despite being relatively straightforward, demonstrate significant improvements over the state of the art. Code and models are available on our webpage.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/1f3cea63caa1c773f8e1676967271dac4166b4c7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Human Motion Diffusion as a Generative Prior (2023, background): “…generation approach [Ho et al. 2020; Sohl-Dickstein et al. 2015] have quickly found their way into the field, yielding motion generation models that produce diverse and high-quality sequences from text or other forms of control [Guo et al. 2022; Petrovich et al. 2022; Tevet et al. 2022, 2023].”
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022, background): “, 2022), motion synthesis (Tevet et al., 2022; Petrovich et al., 2022), style transfer (Kwon & Ye, 2021; Liu et al.”
MotionGPT: Human Motion as a Foreign Language (2023, positive): “Existing text-to-motion generation methods [10, 30, 46, 11, 1, 17] can be characterized as caption-to-motion, where the models take in a pure text description of the desired motion.”
ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model (2023, methodology): “TEMOS [16] employs a Variational Autoencoder (VAE) [11] to replace the deterministic auto-encoder structures.”
EDGE: Editable Dance Generation From Music (2022, background): “Gener-ating human motion conditioned on various inputs—e.g., joystick control [34], class-conditioning [15, 43], text-to-motion [44,63], seed motions [8,22,46,62]—is also an active area of study.”
Human Motion Diffusion Model (2022, methodology): “Motion can be controlled with a high-level guidance given from action class (Guo et al., 2020; Petrovich et al., 2021; Cervantes et al., 2022), audio (Li et al., 2021; Aristidou et al., 2022) and natural language (Ahuja & Morency, 2019; Petrovich et al., 2022).”
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, methodology): “This architecture is also employed in later works [12], [14], [32].”
Executing your Commands via Motion Diffusion in Latent Space (2022, methodology): “Among all conditional modalities, text-based conditional human motion synthesis has been driving and dominating research frontiers because the language descriptors provide a convenient and natural user interface for people to interact with computers [47, 2, 75, 69, 28].”
MoMask: Generative Masked Modeling of 3D Human Motions (2023, methodology): “Recently, we have witnessed the surge of works for neural motion generation, with conditioning on various domains such as motion prefix [29, 33], action class [6, 14, 31, 35], audio [13, 40, 43, 53], texts [9, 15, 16, 36, 42].”
Generating Human Motion from Textual Descriptions with Discrete Representations (2023, methodology): “001 TEMOS § [53] 0 .”