Evolving Space-Time Neural Architectures for Videos

By A. Piergiovanni, Anelia Angelova, Alexander Toshev, Michael S. Ryoo. 2018

62 citations recorded by the source corpus.

We present a new method for finding video CNN architectures that more optimally capture rich spatio-temporal information in videos. Previous work, taking advantage of 3D convolutions, obtained promising results by manually designing CNN video architectures. We here develop a novel evolutionary algorithm that automatically explores models with different types and combinations of layers to jointly learn interactions between spatial and temporal aspects of video representations. We demonstrate the generality of this algorithm by applying it to two meta-architectures. Further, we propose a new component, the iTGM layer, which more efficiently utilizes its parameters to allow learning of space-time interactions over longer time horizons. The iTGM layer is often preferred by the evolutionary algorithm and allows building cost-efficient networks. The proposed approach discovers new diverse and interesting video architectures that were unknown previously. More importantly they are both more accurate and faster than prior models, and outperform the state-of-the-art results on four datasets: Kinetics, Charades, Moments in Time and HMDB. We will open source the code and models, to encourage future model development.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, background): “Despite its advantages, the temporal global average pooling (TGAP) layer which is used at the end of all 3D CNN architectures [1,2,7,12,22,27,28,35] hinders the richness of final temporal information.”
A Comprehensive Survey of Neural Architecture Search (2020, background): “k architecture, and proposes to take device into account for multi-objective optimization tasks. In [32–34], their structure of each cell is the same, only need to search for a cell. For video tasks, [44] uses L ×3 ×3, stride = 1,2,2 max-pooling instead of reduction cell. And, in order to adapt to t”
A Survey on Evolutionary Neural Architecture Search (2020, methodology): “It has been effectively applied to many other methods [70].”
A Multigrid Method for Efficiently Training Video Models (2019, background): “They are currently the state of the art for video understanding [3,9,12,19,30–32, 34,43– 45, 47– Efﬁcient training can also be advanced through, e.g ., optimization methods ( e.g ., [8,24,33,39]), pre-training [3,11], distributed training [13,50], or advances in hardware [22] or software/framework…”
TokenLearner: What Can 8 Learned Tokens Do for Images and Videos? (2021, background): “More advanced network designs have also been extremely popular in video CNNs particularly two-stream ones [6], [43], [48], [49], [50], [51] and, recently, architecture searched ones [27], [33], [37], [38], [52], [53].”
In the Eye of the Beholder: Gaze and Actions in First Person Video (2020, background): “explored, including mixed 2D and 3D networks [45], [46], grouped or depthwise 3D convolutions [24], [47], [48], [49], and the use of network architecture search [50].”