Learning Individual Styles of Conversational Gesture

By Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, Jitendra Malik. Computer Vision and Pattern Recognition, 2019

398 citations recorded by the source corpus.

Human speech is often accompanied by hand and arm gestures. We present a method for cross-modal translation from "in-the-wild" monologue speech of a single speaker to their conversational gesture motion. We train on unlabeled videos for which we only have noisy pseudo ground truth from an automatic pose detection system. Our proposed model significantly outperforms baseline methods in a quantitative comparison. To support research toward obtaining a computational understanding of the relationship between gesture and speech, we release a large video dataset of person-specific gestures.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/e609684188c842092ae7bcae81429471e422df4d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

AI Choreographer: Music Conditioned 3D Dance Generation with AIST++ (2021, methodology): “More recent approach employ LSTMs [5, 79, 90, 97, 42], GANs [51, 78, 28], transformer encoder with RNN de-coder [36] or convolutional [2, 92] sequence-to-sequence models.”
Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory (2022, background): “Most existing dance generation studies intend to solve the two challenges both in a single ingeniously designed network that directly maps music to 3D joint sequence in high-dimensional continuous space [3, 19, 37, 11, 2, 30].”
Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision (2020, background): “Therefore, audiovisual fusion can make full use of complementary information and become an extensive tendency in computer vision and speech recognition communities [1,2,32,26,10,19].”
Human Hands as Probes for Interactive Object Understanding (2021, background): “This understanding can take different forms: grasp type classification [3,51,63] from a hand-defined taxonomy [15], hand keypoint and pose estimation [17], understanding gestures [18], detecting hands, their states and objects of interaction [55, 56], 3D reconstruction of the hand and the object of ”
What comprises a good talking-head video generation?: A Survey and Benchmark (2020, background): “People naturally emit spontaneous motions such as head movements and emotional expressions when they speak, which contain nonverbal information that helps the audience comprehend the speech content [3, 8].”
TEMOS: Generating diverse human motions from textual descriptions (2022, background): “Motion synthesis methods can be broadly divided into two categories: (i) unconstrained generation, which models the entire space of possible motions [51, 56, 58] and (ii) conditioned synthesis, which aims for controllability such as using music [26, 27, 28], speech [7, 14], action [15, 39], and text”
MakeItTalk: Speaker-Aware Talking Head Animation (2020, background): “Recently, Ginosar et al. [2019] predicted conversational gestures and skeleton movements from speech signals.”