Trespassing the Boundaries: Labeling Temporal Bounds for Object Interactions in Egocentric Video

By Davide Moltisanti, Michael Wray, Walterio W. Mayol-Cuevas, Dima Damen. 2017

34 citations recorded by the source corpus.

Manual annotations of temporal bounds for object interactions (i.e. start and end times) are typical training input to recognition, localization and detection algorithms. For three publicly available egocentric datasets, we uncover inconsistencies in ground truth temporal bounds within and across annotators and datasets. We systematically assess the robustness of state-of-the-art approaches to changes in labeled temporal bounds, for object interaction recognition. As boundaries are trespassed, a drop of up to 10% is observed for both Improved Dense Trajectories and Two- Stream Convolutional Neural Network.,, We demonstrate that such disagreement stems from a limited understanding of the distinct phases of an action, and propose annotating based on the Rubicon Boundaries, inspired by a similarly named cognitive model, for consistent temporal bounds of object interactions. Evaluated on a public dataset, we report a 4% increase in overall accuracy, and an increase in accuracy for 55% of classes when Rubicon Boundaries are used for temporal annotations.

Subfields: Computer Vision and Pattern Recognition, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Extraction

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[160] Davide Moltisanti, Michael Wray, Walterio Mayol-Cuevas, and Dima Damen.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, neutral): “overfit to the data and annotations [16,17].”
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset (2018, methodology): “To ensure the annotators are trained to perform temporal localization, we use a clip from our previous work’s understanding [35] that explains temporal bounds of actions.”
EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition (2019, background): “Fusion in Egocentric AR: Late fusion of appearance and motion has been frequently used in egocentric AR [8, 24, 38, 40], as well as extended to additional streams aimed at capturing egocentric cues [21, 37, 38].”
Diagnosing Error in Temporal Action Detectors (2018, methodology): “Inspired by this work, [11,19,27,30,31,32,36,44] provided insightful analysis of algorithms for multiple localization tasks such as human pose estimation, object detection at large-scale and multi-label action classification, in still images in most of the cases.”
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines (2020, background): “standing [41] that explains temporal bounds of actions.”
Action Recognition From Single Timestamp Supervision in Untrimmed Videos (2019, background): “Collecting these boundaries is not only notoriously burdensome and expensive, but also potentially ambiguous and often arbitrary [21, 29, 7].”