Oops! Predicting Unintentional Action in Video
By D. Epstein, Carl Vondrick. 2019
122 citations recorded by the source corpus.
From just a short glance at a video, we can often tell whether a person's action is intentional or not. Can we train a model to recognize this? We introduce a dataset of in-the-wild videos of unintentional action, as well as a suite of tasks for recognizing, localizing, and anticipating its onset. We train a supervised neural network as a baseline and analyze its performance compared to human consistency on the tasks. We also investigate self-supervised representations that leverage natural signals in our dataset, and show the effectiveness of an approach that uses the intrinsic speed of video to perform competitively with highly-supervised pretraining. However, a significant gap between machine and human performance remains.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Kristen Grauman
- Trevor Darrell
- Jianfeng Gao
- Cordelia Schmid
- Zhe Gan
- Devi Parikh
- Abhinav Kumar Gupta
In Computer Vision and Pattern Recognition
- Kristen Grauman
- Luc Van Gool
- Jitendra Malik
- Trevor Darrell
- Andrew Zisserman
- Cordelia Schmid
- Dacheng Tao
- Ming-Hsuan Yang
Citations in the corpus
- Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[58] Dave Epstein, Boyuan Chen, and Carl Vondrick.”
- Self-supervised Co-training for Video Representation Learning (2020, unknown): “motion [1, 14, 31], co-occurrence [30], jigsaw [37], rotation [33], speed prediction [6, 17, 62], future prediction [25, 26, 60], or by temporal coherence [40, 41, 61, 63].”
- RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning (2020, background): “Epstein et al. [9] try to predict whether a clip is sped up or not.”
- What Is More Likely to Happen Next? Video-and-Language Future Event Prediction (2020, background): “, 2017), future action labels (Vondrick et al., 2016a; Gao et al., 2017b; Shi et al., 2018; Epstein et al., 2020), future human motions (Fragkiadaki et al.”
- VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning (2021, background): “In some of these approaches, an objective is constructed around hand-crafted heuristics tasks like spatial arrangement, color, playback speed or frame order predictions [4, 17, 19, 22, 25, 40, 66, 72].”