A Spatio-temporal Transformer for 3D Human Motion Prediction

By Emre Aksan, Peng Cao, Manuel Kaufmann, Otmar Hilliges. International Conference on 3D Vision, 2020

296 citations recorded by the source corpus.

We propose a novel Transformer-based architecture for the task of generative modelling of 3D human motion. Previous work commonly relies on RNN-based models considering shorter forecast horizons reaching a stationary and often implausible state quickly. Recent studies show that implicit temporal representations in the frequency domain are also effective in making predictions for a predetermined horizon. Our focus lies on learning spatio-temporal representations autoregressively and hence generation of plausible future developments over both short and long term. The proposed model learns high dimensional embeddings for skeletal joints and how to compose a temporally coherent pose via a decoupled temporal and spatial self-attention mechanism. Our dual attention concept allows the model to access current and past information directly and to capture both the structural and the temporal dependencies explicitly. We show empirically that this effectively learns the underlying motion dynamics and reduces error accumulation over time observed in auto-regressive models. Our model is able to make accurate short-term predictions and generate plausible motion sequences over long horizons. We make our code publicly available at https://github.com/eth-ait/motion-transformer.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/2b39bcfca5f7dac199632e384692ef669ef2157f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

FaceFormer: Speech-Driven 3D Facial Animation with Transformers (2021, background): “Some of the most recent works on 3D body motion synthesis [1, 4, 46] and 3D dance generation [36, 37, 57] have explored the power of transformer in modeling sequential data and produced impressive results.”
Human Motion Diffusion Model (2022, background): “The transformer architecture is temporally aware, enabling learning arbitrary length motions, and is well-proven for the motion domain (Petro-vich et al., 2021; Duan et al., 2021; Aksan et al., 2021).”