Robust motion in-betweening

By Félix G. Harvey, Mike Yurick, Derek Nowrouzezahrai, C. Pal. ACM Transactions on Graphics, 2020

396 citations recorded by the source corpus.

In this work we present a novel, robust transition generation technique that can serve as a new tool for 3D animators, based on adversarial recurrent neural networks. The system synthesises high-quality motions that use temporally-sparse keyframes as animation constraints. This is reminiscent of the job of in-betweening in traditional animation pipelines, in which an animator draws motion frames between provided keyframes. We first show that a state-of-the-art motion prediction model cannot be easily converted into a robust transition generator when only adding conditioning information about future keyframes. To solve this problem, we then propose two novel additive embedding modifiers that are applied at each timestep to latent representations encoded inside the network's architecture. One modifier is a time-to-arrival embedding that allows variations of the transition length with a single model. The other is a scheduled target noise vector that allows the system to be robust to target distortions and to sample different transitions given fixed keyframes. To qualitatively evaluate our method, we present a custom MotionBuilder plugin that uses our trained model to perform in-betweening in production scenarios. To quantitatively evaluate performance on transitions and generalizations to longer time horizons, we present well-defined in-betweening benchmarks on a subset of the widely used Human3.6M dataset and on LaFAN1, a novel high quality motion capture dataset that is more appropriate for transition generation. We are releasing this new dataset along with this work, with accompanying code for reproducing our baseline results.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/b660ba42a69fa8ccba5fe6f4c153d7a7d933c890

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, background): “Some works focus on proposing appropriate discriminator networks for motion generation to improve the synthesis quality [34], [35], [36].”
Executing your Commands via Motion Diffusion in Latent Space (2022, background): “Human Motion Synthesis allows rich inputs of multimodal data, such as text [16, 47, 69, 18, 2, 28], action category [46, 19], incomplete pose sequences [12, 20, 69], control signals [65, 66, 45], musics [34, 33, 32] and image(s) [55, 9], here, we focus on some typical tasks.”
Human Motion Diffusion Model (2022, methodology): “Others (Harvey & Pal, 2018; Kaufmann et al., 2020; Harvey et al., 2020; Duan et al., 2021) solve in-betweening and super-resolution tasks using bi-directional GRU (Cho et al., 2014) and Transformer (Vaswani et al., 2017) architectures.”
Action-Conditioned 3D Human Motion Synthesis with Transformer VAE (2021, background): “There is a related line of work on motion “in-betweening” that takes both past and future poses and “inpaints” plausible motions between them; see [22] for more.”
BABEL: Bodies, Action and Behavior with English Labels (2021, background): “On the other hand, motion-capture (mocap) datasets [2, 8, 11, 14] are small in scale and are only sparsely labeled with very few actions.”
Generating Human Motion from Textual Descriptions with Discrete Representations (2023, background): “In addition to synthesizing motion conditioning on past motion, another related topic is generating motion “in-betweening” that takes both past and future poses and fills motion between them [18,27,28,35,64].”