RED: Reinforced Encoder-Decoder Networks for Action Anticipation

By Jiyang Gao, Zhenheng Yang, Ramkant Nevatia. 2017

218 citations recorded by the source corpus.

Action anticipation aims to detect an action before it happens. Many real world applications in robotics and surveillance are related to this predictive capability. Current methods address this problem by first anticipating visual representations of future frames and then categorizing the anticipated representations to actions. However, anticipation is based on a single past frame's representation, which ignores the history trend. Besides, it can only anticipate a fixed future time. We propose a Reinforced Encoder-Decoder (RED) network for action anticipation. RED takes multiple history representations as input and learns to anticipate a sequence of future representations. One salient aspect of RED is that a reinforcement module is adopted to provide sequence-level supervision; the reward function is designed to encourage the system to make correct predictions as early as possible. We test RED on TVSeries, THUMOS-14 and TV-Human-Interaction datasets for action anticipation and achieve state-of-the-art performance on all datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[77] Jiyang Gao, Zhenheng Yang, and Ram Nevatia.”
Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, background): “Beside the online action prediction task, the problem of temporal action detection [22, 56, 56–66] also copes with untrimmed videos.”
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, methodology): “Nevertheless, our work builds on some of the ideas explored in past works such as the use of LSTMs [8], [12], [14] to anticipate actions, the use of the encoder-decoder framework to encode past observations and produce hypotheses of future actions [8], and the use of object specific features [56] to”
When will you do what? - Anticipating Temporal Occurrences of Activities (2018, methodology): “This work has been extended in [3], where they use an encoder-decoder network to predict a sequence of future representations based on a history of previous frames.”
What Would You Expect? Anticipating Egocentric Actions With Rolling-Unrolling LSTMs and Modality Attention (2019, methodology): “Additionally, we show that other approaches, such as the direct regression of future representations [18, 58], do not achieve satisfactory performance in the considered scenario.”
Leveraging Uncertainty to Rethink Loss Functions and Evaluation Measures for Egocentric Action Anticipation (2018, background): “[12] proposed an encoder-decoder LSTM model to anticipate future representations and actions.”
Anticipative Video Transformer (2021, background): “While well explored in thirdperson video [2, 26, 37, 38, 46, 48, 79, 87], it has recently gained in popularity for first-person (egocentric) videos [13, 14, 16, 24, 55, 62, 74], due to its applicability on wearable computing platforms.”
Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition (2019, methodology): “For instance, it is utilized for action detection in [53], for action anticipation in [14], for natural language grounding in [19] and for video description in [46].”
SSNet: Scale Selection Network for Online 3D Action Prediction (2018, background): “Beside the online action prediction task, the problem of temporal action detection [68, 73, 48, 33, 7, 53, 68, 14, 54, 16, 15, 65, 82] also copes with untrimmed videos.”
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines (2020, background): “This has been changing recently, with the release of the action classification benchmarks such as [10], [11], [12], [13], [14], [15], [16], [17], [18].”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, methodology): “55) for training and evaluating egocentric action anticipation models in 2018 [37], works on action anticipation mainly considered third-person scenario [47, 9, 7].”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, background): “The task of action anticipation has been investigated from the third person point of view [53, 31, 35, 101].”
Dynamic Sampling Networks for Efficient Action Recognition in Videos (2020, methodology): “[49] used reinforcement learning to train an encoder-decoder network for action anticipation.”