Convolutional Sequence to Sequence Model for Human Dynamics

By Zhen Zhang, Wee Sun Lee, Gim Hee Lee. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

370 citations recorded by the source corpus.

Human motion modeling is a classic problem in computer vision and graphics. Challenges in modeling human motion include high dimensional prediction as well as extremely complicated dynamics.We present a novel approach to human motion modeling based on convolutional neural networks (CNN). The hierarchical structure of CNN makes it capable of capturing both spatial and temporal correlations effectively. In our proposed approach, a convolutional long-term encoder is used to encode the whole given motion sequence into a long-term hidden variable, which is used with a decoder to predict the remainder of the sequence. The decoder itself also has an encoder-decoder structure, in which the short-term encoder encodes a shorter sequence to a short-term hidden variable, and the spatial decoder maps the long and short-term hidden variable to motion predictions. By using such a model, we are able to capture both invariant and dynamic information of human motion, which results in more accurate predictions. Experiments show that our algorithm outperforms the state-of-the-art methods on the Human3.6M and CMU Motion Capture datasets. Our code is available at the project website1.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/575d58c9d74a9a7cc62954c755309c7e37498baf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Qi Tian
2. Mathieu Salzmann
3. Yi Yang
4. Luc Van Gool
5. Ya Zhang
6. Otmar Hilliges
7. Yanfeng Wang
8. Miaomiao Liu

In Control and Systems Engineering

1. Qi Tian
2. Siheng Chen
3. Wei Mao
4. Peng Cao
5. Yangheng Zhao
6. Ya Zhang
7. Yanfeng Wang
8. Otmar Hilliges

Citations in the corpus

• Deep Learning-Based Vehicle Behavior Prediction for Autonomous Driving Applications: A Review (2020, background): “This success motivates researchers in other domains to represent their data as an image to be able to apply CNNs on them [61].”
• Dynamic Multiscale Graph Neural Networks for 3D Skeleton Based Human Motion Prediction (2020, background): “[32], CSM [26], TPRNN [7], AGED [12], and Imit-L [42], or separated bodies e.g. Skel-TNet [14], and Traj-GCN [31].”
• History Repeats Itself: Human Motion Prediction via Motion Attention (2020, methodology): “3 convSeq2Seq [17] 17.”
• A Spatio-temporal Transformer for 3D Human Motion Prediction (2020, methodology): “Convolutional neural networks (CNN) in a sequence-to-sequence framework have also been proposed [13, 21, 23].”