Neural state machine for character-scene interactions

By Sebastian Starke, He Zhang, Taku Komura, Jun Saito. ACM Transactions on Graphics, 2019

366 citations recorded by the source corpus.

We propose Neural State Machine, a novel data-driven framework to guide characters to achieve goal-driven actions with precise scene interactions. Even a seemingly simple task such as sitting on a chair is notoriously hard to model with supervised learning. This difficulty is because such a task involves complex planning with periodic and non-periodic motions reacting to the scene geometry to precisely position and orient the character. Our proposed deep auto-regressive framework enables modeling of multi-modal scene interaction behaviors purely from data. Given high-level instructions such as the goal location and the action to be launched there, our system computes a series of movements and transitions to reach the goal in the desired state. To allow characters to adapt to a wide range of geometry such as different shapes of furniture and obstacles, we incorporate an efficient data augmentation scheme to randomly switch the 3D geometry while maintaining the context of the original motion. To increase the precision to reach the goal during runtime, we introduce a control scheme that combines egocentric inference and goal-centric inference. We demonstrate the versatility of our model with various scene interaction tasks such as sitting on a chair, avoiding obstacles, opening and entering through a door, and picking and carrying objects generated in real-time just from a single model.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/68f50216374c50382e17890aa25871230517af13

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

BABEL: Bodies, Action and Behavior with English Labels (2021, background): “Modeling the likelihood of simultaneous actions and action transitions has applications in reasoning about action affordances in robotics and virtual avatars, motion synthesis [33], activity forecasting [16], animation [31], and action recognition.”
Diffusion-based Generation, Optimization, and Planning in 3D Scenes (2023, background): “Several applications have emerged on conditional scene generation [24,47,71,81,88], human pose [16,32,87,90,92] and motion generation [14, 32, 49, 60, 68–70, 73] in Physics-based Optimization in 3D Scenes Producing physically plausible generations compatible with 3D scenes is challenging in the…”
Action-Conditioned 3D Human Motion Synthesis with Transformer VAE (2021, background): “Most relevant here are the phase-functioned neural networks [26] and neural state machines [57].”
GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping (2021, methodology): “We empirically observe that our combination of inputs and outputs, inspired from work on character control [56], leads to more realistic results.”
TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts (2022, background): “Meanwhile, [19,18,40] attempts to model human motions according to instant control signals such as velocity and directional readouts.”
AMP (2021, background): “When provided with sufficiently large and high-quality datasets, kinematic methods are able to produce realistic motions for a large variety of sophisticated skills [Agrawal and van de Panne 2016; Lee et al. 2018, 2010b; Levine et al. 2011; Starke et al. 2019].”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, background): “Existing methods [32, 68] however are learned from high quality curated data captured using optical marker based motion capture systems or wearable sensors.”
Catch & Carry (2019, background): “Recent kinematic-based approaches show impressive interactions between a humanoid and the environment [Starke et al. 2019], but this requires modeling the motion as an explicit function of a high-dimensional volumetric scene context.”
Executing your Commands via Motion Diffusion in Latent Space (2022, methodology): “Human Motion Synthesis allows rich inputs of multimodal data, such as text [16, 47, 69, 18, 2, 28], action category [46, 19], incomplete pose sequences [12, 20, 69], control signals [65, 66, 45], musics [34, 33, 32] and image(s) [55, 9], here, we focus on some typical tasks.”
AI Choreographer: Music Conditioned 3D Dance Generation with AIST++ (2021, background): “Phase-functioned neural networks and it’s variations [94, 33, 73, 74] address this issue via conditioning the network weights on phase, however, they do not scale well to represent a wide variety of motion.”
Robust motion in-betweening (2020, background): “Online control from a gamepad has also been tackled with phase-aware [Holden et al. 2017], mode-aware [Zhang et al. 2018] and action-aware [Starke et al. 2019] neural networks that can automatically choose a mixture of network weights at run-time to disambiguate possible motions.”
HuMoR: 3D Human Motion Model for Robust Pose Estimation (2021, background): “[86] Sebastian Starke, He Zhang, Taku Komura, and Jun Saito.”
Artificial intelligence in the creative industries: a review (2020, methodology): “rning models of motion from captured real motion sequences. These motion characteristics can be learnt using DL-based approaches, such as autoencoders [47], LSTMs [53], and motion prediction networks [48]. Then, the inference applies these characteristics from the trained model to animate characters”
GRAB: A Dataset of Whole-Body Human Grasping of Objects (2020, methodology): “[63] use Xsens IMU sensors [75] to capture the main body of a subject interacting with large objects, and learn to synthesize avatar motion in virtual worlds.”