Language2Pose: Natural Language Grounded Pose Forecasting

By Chaitanya Ahuja, Louis-philippe Morency. International Conference on 3D Vision, 2019

380 citations recorded by the source corpus.

Generating animations from natural language sentences finds its applications in a a number of domains such as movie script visualization, virtual human animation and, robot motion planning. These sentences can describe different kinds of actions, speeds and direction of these actions, and possibly a target destination. The core modeling challenge in this language-to-pose application is how to map linguistic concepts to motion animations. In this paper, we address this multimodal problem by introducing a neural architecture called Joint Language-to-Pose (or JL2P), which learns a joint embedding of language and pose. This joint embedding space is learned end-to-end using a curriculum learning approach which emphasizes shorter and easier sequences first before moving to longer and harder ones. We evaluate our proposed model on a publicly available corpus of 3D pose data and human-annotated sentences. Both objective metrics and human judgment evaluation confirm that our proposed approach is able to generate more accurate animations and are deemed visually more representative by humans than other data driven approaches.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/967b2d10b8b378f1da43fd4d9107826e540e1112

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model (2023, background): “JL2P [1] attempts to create a joint embedding space by applying the same reconstruction task on both text and motion embedding.”
Action2Motion: Conditioned Generation of 3D Human Motions (2020, background): “o be inferred from text, an emerging subject thatismorerelatedtoourproblem.Pioneereffortssuchas[1,18,24] mainly resort to encoder-decoder RNN architecture for languageto-pose translation. The work of [2] learns a joint embedding space between sentences and human pose sequences. More recently, [26] a”
MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, methodology): “We compare our model to JL2P [Ahuja and Morency 2019].”
Executing your Commands via Motion Diffusion in Latent Space (2022, methodology): “Two typical methods address this problem: 1) the cross-modal compatible latent space between motion and language [47, 2] and 2) the conditional diffusion model [74, 68, 28].”
Human Motion Diffusion Model (2022, methodology): “JL2P (Ahuja & Morency, 2019) learns the KIT motion-language dataset (Plappert et al., 2016) with an auto-encoder, limiting one-to-one mapping from text to motion.”
Speech gesture generation from the trimodal context of text, audio, and speaker identity (2020, methodology): “A study by (Ahuja and Morency 2019) represented both human motion and descriptive text as vectors in the same embedding space.”
MotionGPT: Human Motion as a Foreign Language (2023, neutral): “Existing text-to-motion generation methods [10, 30, 46, 11, 1, 17] can be characterized as caption-to-motion, where the models take in a pure text description of the desired motion.”
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, background): “MotionDiffuse excels at modeling complicated data distribution and generating vivid motion sequences.”
BABEL: Bodies, Action and Behavior with English Labels (2021, neutral): “This includes simultaneous actions (nested brackets) and transitions between actions (shaded gray box). tion [28, 41], and generating human movement that is conditioned on semantics, e.g., motion synthesis conditioned on actions [12], or sentences [3, 19].”
Generating Human Motion from Textual Descriptions with Discrete Representations (2023, background): “[20] learn two manifold representations for the upper body and the lower body movements, which shows improved performance compared to Language2Pose [3].”
Generating Diverse and Natural 3D Human Motions from Text (2022, methodology): “We compare our work to three state-of-the-art methods: Seq2Seq [21], Language2Pose [1] and Text2Gesture [4].”
Action-Conditioned 3D Human Motion Synthesis with Transformer VAE (2021, background): “Text2Action [1] and Language2Pose [2] similarly explore conditioning the motion generation on textual descriptions.”
TEMOS: Generating diverse human motions from textual descriptions (2022, methodology): “Others build joint cross-modal embeddings to map the text and motion to the same space [2, 13, 50], which has been a success in other research area [5, 42, 52, 53].”
PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, methodology): “These methods often use GANs or VAEs to generate motions from various conditions such as past motions [3, 7, 87, 91, 92], key frames [24], music [37, 38, 100], text [1, 9, 22, 62], and action labels [10, 23, 61].”
MoMask: Generative Masked Modeling of 3D Human Motions (2023, background): “Early works [1, 12, 20, 27, 38] commonly model motion generation deterministically, resulting in averaged and blurry motion results.”
TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts (2022, methodology): “Furthermore, our proposed approach is able to explicitly model the underlying distribution of 3D motions conditioned on texts, instead of regressing the mean motions as in previous works [2,11,26,56,37], thus allows non-deterministic text2motion generation.”