Omni-sourced Webly-supervised Learning for Video Recognition

By Haodong Duan, Yue Zhao, Yuanjun Xiong, Wentao Liu, Dahua Lin. 2020

105 citations recorded by the source corpus.

We introduce OmniSource, a novel framework for leveraging web data to train video recognition models. OmniSource overcomes the barriers between data formats, such as images, short videos, and long untrimmed videos for webly-supervised learning. First, data samples with multiple formats, curated by task-specific data collection and automatically filtered by a teacher model, are transformed into a unified form. Then a joint-training strategy is proposed to deal with the domain gaps between multiple data sources and formats in webly-supervised learning. Several good practices, including data balancing, resampling, and cross-dataset mixup are adopted in joint training. Experiments show that by utilizing data from multiple sources and formats, OmniSource is more data-efficient in training. With only 3.5M images and 800K minutes videos crawled from the internet without human labeling (less than 2% of prior works), our models learned with OmniSource improve Top-1 accuracy of 2D- and 3D-ConvNet baseline models by 3.0% and 3.9%, respectively, on the Kinetics-400 benchmark. With OmniSource, we establish new records with different pretraining strategies for video recognition. Our best models achieve 80.4%, 80.5%, and 83.6 Top-1 accuracies on the Kinetics-400 benchmark respectively for training-from-scratch, ImageNet pre-training and IG-65M pre-training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Liu
2. Qiuhong Ke
3. Hossein Rahmani
4. Zehua Sun
5. Limin Wang
6. Tianhao Li
7. G. Wang
8. Amir Shahroudy

In Computer Vision and Pattern Recognition

1. Jun Liu
2. Qiuhong Ke
3. Hossein Rahmani
4. Zehua Sun
5. G. Wang
6. Tianhao Li
7. Limin Wang
8. Amir Shahroudy

Citations in the corpus

• Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “based HAR methods can leverage large-scale web videos to pre-train models for better recognition performance [143], [144].”
• Learning Spatiotemporal Features via Video and Text Pair Discrimination (2020, methodology): “pendix. Instagram-300k. To avoid data bias in Kinetics caused by human annotation (i.e., trimmed videos with an action), we further verify the eectiveness our CPD model on a purely web video dataset [11]. This new dataset is constructed from Instagram by searching action label of Kinetics-400 but wi”