High Five: Recognising human interactions in TV shows

By Alonso Patron-Perez, Marcin Marszalek, Andrew Zisserman, Ian David Reid. 2010

197 citations recorded by the source corpus.

In this paper we address the problem of recognising interactions between two people in realistic scenarios for video retrieval purposes. We develop a per-person descriptor that uses attention (head orientation) and the local spatial and temporal context in a neighbourhood of each detected person. Using head orientation mitigates camera view ambiguities, while the local context, comprised of histograms of gradients and motion, aims to capture cues such as hand and arm movement. We also employ structured learning to capture spatial relationships between interacting individuals. We train an initial set of one-vs-the-rest linear SVM classifiers, one for each interaction, using this descriptor. Noting that people generally face each other while interacting, we learn a structured SVM that combines head orientation and the relative location of people in a frame to improve upon the initial classification obtained with our descriptor. To test the efficacy of our method, we have created a new dataset of realistic human interactions comprised of clips extracted from TV shows, which represents a very difficult challenge. Our experiments show that using structured learning improves the retrieval results compared to using the interaction classifiers independently.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “Table 4 also compares EPIC-KITCHENS-100 with other datasets used for action anticipation [30,54,55,56,61,62,40].”
Anticipating the future by watching unlabeled video (2015, methodology): “To evaluate how well our method can forecast actions, we use the TV Human Interactions dataset [26], which consists of people performing four different actions (hand shake, high five, hug, and kissing).”
A database for fine grained activity detection of cooking activities (2012, background): “The Coffee and Cigarettes [17], High Five [24] are different to the other movie datasets by promoting activity detection rather than classification.”
The Action Similarity Labeling Challenge (2012, methodology): “Other data sets employing videos from such sources are the data set made available in [28], which includes actions extracted from a TV series, the work in [11], which classifies actions in broadcast sports videos, and the recent work in [15], which explores human interactions in TV shows.”
Human Instance Segmentation from Video using Detector-based Conditional Random Fields (2011, background): “Ever since computer vision researchers embarked on developing algorithms for complete scene understanding, understanding human activities has been an important goal [5, 17].”
A Critical Review of Action Recognition Benchmarks (2013, methodology): “High-Five [26] 2010 4 300 23 different TV shows 30-600 frames, realistic human interactions: varying number of actors, scale, and views 68% [18]”
LAEO-Net: Revisiting People Looking at Each Other in Videos (2019, methodology): “First, we introduce two new datasets: UCO-LAEO and AVA-LAEO, and then, two other datasets: AFLW [12], and TVHID [22].”
Anticipating Visual Representations from Unlabeled Video (2015, methodology): “We use the TV Human Interactions dataset [25] because it satisﬁes these requirements.”
RED: Reinforced Encoder-Decoder Networks for Action Anticipation (2017, methodology): “TV-Human-Interaction [13] dataset contains 300 video clips extracted from 23 different TV shows.”
Video-Based Human Behavior Understanding: A Survey (2013, methodology): “For close-range interactions, the TV Human interactions dataset (TV HID) [168] is used to analyze interactions occurring in TV footages [170].”
On Recognizing Actions in Still Images via Multiple Features (2012, background): “In [18], it has been shown that facial features can be useful in interaction recognition, and here we investigate their effect to generic actions.”
A Hybrid Loss for Multiclass and Structured Prediction (2014, methodology): “To compute r, we employ a simple method in [14] which only requires the bounding boxes of person i and j.”
A Robust and Efficient Video Representation for Action Recognition (2015, unknown): “The High Five dataset (Patron-Perez et al, 2010) consists of 300 video clips extracted from 23 different TV shows.”
Ieee Transactions on Pattern Analysis an Machine Intelligence Structured Learning of Human Interactions in Tv Shows (result): “This paper is an extension of our previous work [5].”
Sympathy for the Details: Dense Trajectories and Hybrid Classification Architectures for Action Recognition (2016, methodology): “As in [67,68,5,6], we report mAP for the positive classes (mAP+) using the train/test split provided by the dataset authors.”
Social interactions: A first-person perspective (2012, methodology): “In addition, we compute the orientation (yaw, pitch, roll) of every detected face!.”