Time-Agnostic Prediction: Predicting Predictable Video Frames

By Dinesh Jayaraman, Frederik D. Ebert, Alexei A. Efros, Sergey Levine. 2018

94 citations recorded by the source corpus.

Prediction is arguably one of the most basic functions of an intelligent system. In general, the problem of predicting events in the future or between two waypoints is exceedingly difficult. However, most phenomena naturally pass through relatively predictable bottlenecks---while we cannot predict the precise trajectory of a robot arm between being at rest and holding an object up, we can be certain that it must have picked the object up. To exploit this, we decouple visual prediction from a rigid notion of time. While conventional approaches predict frames at regularly spaced temporal intervals, our time-agnostic predictors (TAP) are not tied to specific times so that they may instead discover predictable "bottleneck" frames no matter when they occur. We evaluate our approach for future and intermediate frame prediction across three robotic manipulation tasks. Our predictions are not only of higher visual quality, but also correspond to coherent semantic subgoals in temporally extended tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Self-Supervised Learning of Video-Induced Visual Invariances (2019, neutral): “The ambiguity in single-frame prediction has been side-stepped via time-agnostic prediction [29], motion segmentation [47], cross-pixel matching [38], and by giving the model a motion cue as input [70].”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “The nodes, termed pivotal states, are the most critical states in recovering action trajectories (Chatzigiorgaki & Skodras, 2009; Jayaraman et al., 2018; Ghosh et al., 2018).”
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations (2020, methodology): “A way to achieve the compositionality we are aiming for is to segment the demonstrations and recompose them, by leveraging methods such as TAPS [18], or a similar approach by Kipf et al.”
CompILE: Compositional Imitation Learning and Execution (2018, background): “Sequence prediction models with adaptive step size (Neitz et al., 2018; Jayaraman et al., 2018) can provide segment boundaries as well, but do not directly learn a policy or latent encodings.”
On the link between conscious function and general intelligence in humans and machines (2022, background): “There are multiple approaches to this: for example, by focusing on bottleneck states (Neitz et al., 2018; Jayaraman et al., 2018).”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, background): “Video anticipation aims to forecast future events in videos, including future frames prediction [9,36,44,53, 80,84,89], action anticipation [25,26,28,58,72,87], and dynamics learning [23,29,67].”
Learning Temporal Dynamics from Cycles in Narrated Video (2021, unknown): “Baselines: We evaluate purely cross-modal features (Section 3.2), given by frozen embedding nets Ψ V , Ψ T , and also use these features as prediction targets for RA and TAP below.”
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation (2019, methodology): “Success rate for navigation tasks using no sub-goals, HVF subgoals, TAP (Jayaraman et al., 2019), and ground truth hand speciﬁed subgoals.”
Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning (2019, methodology): “6.1), a bottleneck-based subgoal predictor (time-agnostic prediction or TAP, Jayaraman et al. (2019)), and subgoals selected at ﬁxed intervals from sequences generated by CIGAN, an alternative sequence modeling approach (Wang et al. (2019)).”
Procedure Planning in Instructional Videos (2019, result): “In contrast to the videos studied in previous works [12, 14], the instructional videos we use in this paper is extremely challenging to generate using visual generator [14].”
KeyIn: Discovering Subgoal Structure with Keyframe-based Video Prediction (2019, methodology): “For planning, we also compare to TAP (Jayaraman et al., 2019) and the low-level planner that plans directly to the goal.”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “Alternatively, a number of recent works have studied long horizon visual planning [52, 53, 73, 74, 75, 50, 76, 77, 78], often by combining a structured search approach with learned components like generative models [73, 50, 76] or distance functions [53, 34].”