A Multigrid Method for Efficiently Training Video Models
By Chaoxia Wu, Ross B. Girshick, Kaiming He, Christoph Feichtenhofer, Philipp Krahenbuhl. 2019
102 citations recorded by the source corpus.
Training competitive deep video models is an order of magnitude slower than training their counterpart image models. Slow training causes long research cycles, which hinders progress in video understanding research. Following standard practice for training image models, video model training has used a fixed mini-batch shape: a specific number of clips, frames, and spatial size. However, what is the optimal shape? High resolution models perform well, but train slowly. Low resolution models train faster, but are less accurate. Inspired by multigrid methods in numerical optimization, we propose to use variable mini-batch shapes with different spatial-temporal resolutions that are varied according to a schedule. The different shapes arise from resampling the training data on multiple sampling grids. Training is accelerated by scaling up the mini-batch size and learning rate when shrinking the other dimensions. We empirically demonstrate a general and robust grid schedule that yields a significant out-of-the-box training speedup without a loss in accuracy for different models (I3D, non-local, SlowFast), datasets (Kinetics, Something-Something, Charades), and training settings (with and without pre-training, 128 GPUs or 1 GPU). As an illustrative example, the proposed multigrid method trains a ResNet-50 SlowFast network 4.5x faster (wall-clock time, same hardware) while also improving accuracy (+0.8% absolute) on Kinetics-400 compared to baseline training. Code is available online.
Subfields: Computer Vision and Pattern Recognition, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
In Optimization & Training
Citations in the corpus
- 3D CNNs with Adaptive Temporal Feature Resolutions (2020, unknown): “Following [27], we use a R50-SlowFast model pre-trained on Kinetics-400 with 64 frames for the fast pathway, speed ratio of α = 4, and channel ratio β = 1/8.”
- On the Transformer Growth for Progressive BERT Training (2020, methodology): “Inspired by the idea of multigrid training in the vision domain (Wu et al., 2020), we train the model with “low-resolution text” through embedding pooling over unmasked tokens.”