Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning
By Zhaowei Cai, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Zhuowen Tu, Stefano Soatto. 2021
165 citations recorded by the source corpus.
We present a plug-in replacement for batch normalization (BN) called exponential moving average normalization (EMAN), which improves the performance of existing student-teacher based self- and semi-supervised learning techniques. Unlike the standard BN, where the statistics are computed within each batch, EMAN, used in the teacher, updates its statistics by exponential moving average from the BN statistics of the student. This design reduces the intrinsic cross-sample dependency of BN and enhances the generalization of the teacher. EMAN improves strong baselines for self-supervised learning by 4-6/1-2 points and semi-supervised learning by about 7/2 points, when 1%/10% supervised labels are available on ImageNet. These improvements are consistent across methods, network architectures, training duration, and datasets, demonstrating the general effectiveness of this technique. The code will be made available online.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Elisa Ricci
- Enrico Fini
- Moin Nabi
- Zhiqi Kang
- Alahari Karteek
- Christopher Kanan
- Tyler L. Hayes
- Ali Ayub
In Computer Vision and Pattern Recognition
- Elisa Ricci
- Enrico Fini
- Moin Nabi
- Zhiqi Kang
- Alahari Karteek
- Luc Van Gool
- Andrew Zisserman
- Andrea Vedaldi
Citations in the corpus
- A soft nearest-neighbor framework for continual semi-supervised learning (2022, positive): “For instance, self-supervised pre-training was found beneficial in [11], exponential moving average normalization was adopted in [11] and contrastive learning was considered in [3].”
- Omni-DETR: Omni-Supervised Object Detection with Transformers (2022, unknown): “This EMA updated teacher can be seen as a temporal ensemble of student models along the training trajectories, which makes it more robust and able to generate more accurate pseudo-labels [1, 5, 21].”