AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures

By Michael S. Ryoo, A. Piergiovanni, Mingxing Tan, Anelia Angelova. 2019

110 citations recorded by the source corpus.

Learning to represent videos is a very challenging task both algorithmically and computationally. Standard video CNN architectures have been designed by directly extending architectures devised for image understanding to include the time dimension, using modules such as 3D convolutions, or by using two-stream design to capture both appearance and motion in videos. We interpret a video CNN as a collection of multi-stream convolutional blocks connected to each other, and propose the approach of automatically finding neural architectures with better connectivity and spatio-temporal interactions for video understanding. This is done by evolving a population of overly-connected architectures guided by connection weight learning. Architectures combining representations that abstract different input types (i.e., RGB and optical flow) at multiple temporal resolutions are searched for, allowing different types or sources of information to interact with each other. Our method, referred to as AssembleNet, outperforms prior approaches on public video datasets, in some cases by a great margin. We obtain 58.6% mAP on Charades and 34.27% accuracy on Moments-in-Time.

Subfields: Computer Vision and Pattern Recognition, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Cordelia Schmid
3. Kristen Grauman
4. Andrew Zisserman
5. Dacheng Tao
6. Trevor Darrell
7. Jitendra Malik
8. Xiaogang Wang

In Neural Architecture Search & AutoML

1. Mingxing Tan
2. Wei Hua
3. Xiaofang Wang
4. Cordelia Schmid
5. Kris M. Kitani

Citations in the corpus

• A Comprehensive Survey of Neural Architecture Search (2020, neutral): “The early NAS [11, 12, 15, 16] and much of the subsequent work [17, 38–40] was to search the network architecture from scratch.”
• MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, methodology): “On the one hand, 3D convolutional neural networks (CNNs) [65, 69, 19, 18, 52] offer state-of-the-art accuracy, but consume copious amounts of memory and computation.”
• AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification (2020, methodology): “AssembleNet [23] proposes to search for the connectivity between multi-stream convolutional blocks for video classification.”
• Attention Bottlenecks for Multimodal Fusion (2021, background): “[54] Michael S Ryoo, AJ Piergiovanni, Mingxing Tan, and Anelia Angelova.”