MARS: Motion-Augmented RGB Stream for Action Recognition

By Nieves Crasto, Philippe Weinzaepfel, Alahari Karteek, Cordelia Schmid. 2019

268 citations recorded by the source corpus.

Most state-of-the-art methods for action recognition consist of a two-stream architecture with 3D convolutions: an appearance stream for RGB frames and a motion stream for optical flow frames. Although combining flow with RGB improves the performance, the cost of computing accurate optical flow is high, and increases action recognition latency. This limits the usage of two-stream approaches in real-world applications requiring low latency. In this paper, we introduce two learning approaches to train a standard 3D CNN, operating on RGB frames, that mimics the motion stream, and as a result avoids flow computation at test time. First, by minimizing a feature-based loss compared to the Flow stream, we show that the network reproduces the motion stream with high fidelity. Second, to leverage both appearance and motion information effectively, we train with a linear combination of the feature-based loss and the standard cross-entropy loss for action recognition. We denote the stream trained using this combined loss as Motion-Augmented RGB Stream (MARS). As a single stream, MARS performs better than RGB or Flow alone, for instance with 72.7% accuracy on Kinetics compared to 72.0% and 65.6% with RGB and Flow streams respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, methodology): “However, SGD is preferred for 3D CNN architectures [11,1,7,27,3].”
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition (2021, methodology): “Our proposed AdaMML framework is also related to prior works in joint appearance and motion modeling [43, 31, 10] that focuses on combining RGB and optical flow streams.”
In the Eye of the Beholder: Gaze and Actions in First Person Video (2020, background): “More recent works focused on learning video representations [52], [53], [54], [55], [56], [57] (including ours [58]), and the modeling of long-range dependencies in videos [59], [60], [61].”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “[125] transferred the knowledge of a teacher model trained on the optical flows to a 3D CNN stu-”
Cross-Domain First Person Audio-Visual Action Recognition through Relative Norm Alignment (2021, methodology): “As shown in [13], the use of optical flow limits the application of several methods in online scenarios, pushing the community either to investigate alternative paths [27, 8] or towards single-stream architectures [63, 13, 29, 52, 44].”