Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition

By Wenhao Wu, Dongliang He, Xiao Tan, Shifeng Chen, Shilei Wen. 2019

135 citations recorded by the source corpus.

Video Recognition has drawn great research interest and great progress has been made. A suitable frame sampling strategy can improve the accuracy and efficiency of recognition. However, mainstream solutions generally adopt hand-crafted frame sampling strategies for recognition. It could degrade the performance, especially in untrimmed videos, due to the variation of frame-level saliency. To this end, we concentrate on improving untrimmed video classification via developing a learning-based frame sampling strategy. We intuitively formulate the frame sampling procedure as multiple parallel Markov decision processes, each of which aims at picking out a frame/clip by gradually adjusting an initial sampling. Then we propose to solve the problems with multi-agent reinforcement learning (MARL). Our MARL framework is composed of a novel RNN-based context-aware observation network which jointly models context information among nearby agents and historical states of a specific agent, a policy network which generates the probability distribution over a predefined action space at each step and a classification network for reward calculation as well as final recognition. Extensive experimental results show that our MARL-based scheme remarkably outperforms hand-crafted strategies with various 2D and 3D baseline methods. Our single RGB model achieves a comparable performance of ActivityNet v1.3 champion submission with multi-modal multi-model fusion and new state-of-the-art results on YouTube Birds and YouTube Cars.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “For example, the frames are first sampled uniformly in [189], [190], and discrete decisions are made for each selected frame to go forward or backward step by step.”
2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition (2020, methodology): “2, our method is orthogonal and thus could be complementary to the line of clip selection methods [41, 43, 20, 12] for efficient video recognition.”
Location-Aware Graph Convolutional Networks for Video Question Answering (2020, background): “In video QA task, understanding untrimmed videos (Zeng et al. 2019a; Wu et al. 2019) is important.”
Dynamic Inference: A New Approach Toward Efficient Video Action Recognition (2020, background): “Action recognition has drawn great research attention in the community [29, 33, 23, 30, 6, 26, 1, 34, 22, 31, 27, 28, 35, 2, 12, 20].”
X3D: Expanding Architectures for Efficient Video Recognition (2020, result): “[1,36,52,73,74,78], which we think can be complementary to our approach.”
AR-Net: Adaptive Frame Resolution for Efficient Action Recognition (2020, unknown): “MultiAgent [59] LiteEval [61] AdaFrame10 [62] AdaFrame5 [62] ListenToLook(MN|R) [20] ListenToLook(IA|R) [20] ListenToLook(IA|IA) [20] SCSampler [33] Ours (ResNet) Ours (EfficientNet)”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, methodology): “MARL uses a ResNet-101 as the recognition network combined with a fully connected layer as a policy network.”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, background): “However, RL policy gradients are often complex, unwieldy to train and require techniques to reduce variance during training as well as it is slow to converge in many applications [58, 59, 23, 57].”
ActionSpotter: Deep Reinforcement Learning Framework for Temporal Action Spotting in Videos (2020, methodology): “Moreover, our framework balances the trade-off between accuracy and frames skipping and, in opposition to [5], [16], [17], [14], ActionSpotter processes videos online.”
Listen to Look: Action Recognition by Previewing Audio (2019, methodology): “Without using guidance from strong human supervision, they ease the learning process by restricting the agent to a rigid action space [14], guiding the selection process of the agent with a global memory module [71], or using multiple agents to collaboratively perform frame selection [69].”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “More-over, unlike existing works, our framework requires neither complex RL policy gradients [61, 58, 64] nor additional modalities such as audio [14, 30] to learn dynamic policies.”
HMS: Hierarchical Modality Selection for Efficient Video Recognition (2021, methodology): “Specifically, AdaFrame and MARL learn to produce dynamic frame/clip usage policies such that fewer time steps are sampled from videos for efficient inference, and IMGAUD2VID further proposes to use audio information to efficiently preview videos and quickly reject non-informative clips instead of us”
Adaptive Focus for Efficient Video Recognition (2021, methodology): “In this subsection, AdaFocus is compared with several competitive baselines that focus on facilitating efficient video recognition, including MultiAgent [42], SCSampler [25], LiteEval [45], AdaFrame [44], Listen-tolook [16] and AR-Net [29].”
Dynamic Sampling Networks for Efficient Action Recognition in Videos (2020, unknown): “5% MARL [64] SEResNeXt152 Kinetics 25 85.”