ECO: Efficient Convolutional Network for Online Video Understanding

By Mohammadreza Zolfaghari, Kamaljeet Singh, Thomas Brox. European Conference on Computer Vision, 2018

529 citations recorded by the source corpus.

The state of the art in video understanding suffers from two problems: (1) The major part of reasoning is performed locally in the video, therefore, it misses important relationships within actions that span several seconds. (2) While there are local methods with fast per-frame processing, the processing of the whole video is not efficient and hampers fast video retrieval or online classification of long-term activities. In this paper, we introduce a network architecture (https://github.com/mzolfaghari/ECO-efficient-video-understanding) that takes long-term content into account and enables fast per-video processing at the same time. The architecture is based on merging long-term content already in the network rather than in a post-hoc fusion. Together with a sampling strategy, which exploits that neighboring frames are largely redundant, this yields high-quality action classification and video captioning at up to 230 videos per second, where each video can consist of a few hundred frames. The approach achieves competitive performance across all datasets while being 10\(\times \) to 80\(\times \) faster than state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/aa63893b34f523973d0692dc74ff22512daac322

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Listen to Look: Action Recognition by Previewing Audio (2019, background): “Some work balances the accuracy-efficiency trade-off by using compressed video representations [68, 49] or designing efficient network architectures [72, 81, 10, 57].”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, background): “Extensive studies have been conducted to remedy this issue by designing efficient and light-weight architectures [35, 11, 43, 36, 53, 59, 42, 28, 31, 10].”
When Video Classification Meets Incremental Classes (2021, background): “, a 2D spatial convolution and a 1D temporal convolution) [37, 45, 59], or mix 2D and 3D convolutions in a single network [48, 59, 65, 66].”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “For example, the ECO architecture [115] uses 2D CNNs to extract spatial features, which are stacked and then fed to”
AdaFrame: Adaptive Frame Selection for Fast Video Recognition (2018, methodology): “Only a few methods consider efficient video classification [38, 31, 40, 20, 10].”
StNet: Local and Global Spatial-Temporal Modeling for Action Recognition (2018, background): “Recently, there are many frameworks proposed to improve 3D convolution (Zolfaghari, Singh, and Brox 2018; Wang et al. 2017a; Xie et al. 2018; Tran et al. 2018; Wang et al. 2017b; Chen et al. 2018).”
Dynamic Sampling Networks for Efficient Action Recognition in Videos (2020, background): “ECO [46] greatly improved recognition speed by manually designing the stacking of sample frames.”
TAM: Temporal Adaptive Module for Video Recognition (2020, unknown): “7% ECO [47] BNIncep+Res18 K400 16f 64G 41.”
VA-RED2: Video Adaptive Redundancy Reduction (2021, background): “Previous works have shown reductions in computational complexity by using hybrid 2D-3D architectures (Xie et al., 2018; Zhou et al., 2018c; Zolfaghari et al., 2018), group convolution (Tran et al., 2019) or selecting salient clips (Korbar et al., 2019).”
STM: SpatioTemporal and Motion Encoding for Action Recognition (2019, methodology): “To handle the heavy computation of 3D CNNs, several methods are proposed to ﬁnd the trade-off between precision and speed [28, 37, 42, 41, 25, 20].”
Grouped Spatial-Temporal Aggregation for Efficient Action Recognition (2019, methodology): “Our method outperforms the late fusion method TRN [35] and ECO [37] because it can better encode the spatial and temporal features.”
2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition (2020, background): “Extensive studies have been conducted on designing efficient network architectures for video recognition [50, 5, 35, 8, 49, 24, 34].”
TDN: Temporal Difference Networks for Efficient Action Recognition (2020, methodology): “Our proposed TDN achieves the best trade-off between accuracy and efficiency, when compared with previous methods such as NL I3D [40], ECO [46], TSM [19] and TEINet [20].”
TSM: Temporal Shift Module for Efficient Video Understanding (2018, methodology): “ECO [61] also uses a similar top-heavy architecture to achieve a very efficient framework.”
Exploiting Spatial-Temporal Modelling and Multi-Modal Fusion for Human Action Recognition (2018, unknown): “As a remedy for the aforementioned drawback, multiple 3D ConvNet models are invented for end-to-end spatial-temporal modelling such as TResNet [6], P3D [9], ECO [16], ARTNet [17] and S3D [18].”
Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition (2019, background): “…29, 35] which extract spatial feature vectors from video frames and then integrate the obtained local feature sequence into one compact video descriptor for recognition; (2) the 2D [45, 36, 10, 7] or 3D convolution based [41, 11, 3, 52, 33, 47, 42, 44, 56, 5] end-to-end video classiﬁcation methods.”