TaskNorm: Rethinking Batch Normalization for Meta-Learning

By John F. Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, Richard E. Turner. 2020

93 citations recorded by the source corpus.

Modern meta-learning approaches for image classification rely on increasingly deep networks to achieve state-of-the-art performance, making batch normalization an essential component of meta-learning pipelines. However, the hierarchical nature of the meta-learning setting presents several challenges that can render conventional batch normalization ineffective, giving rise to the need to rethink normalization in this setting. We evaluate a range of approaches to batch normalization for meta-learning scenarios, and develop a novel approach that we call TaskNorm. Experiments on fourteen datasets demonstrate that the choice of batch normalization has a dramatic effect on both classification accuracy and training time for both gradient based and gradient-free meta-learning approaches. Importantly, TaskNorm is found to consistently improve performance. Finally, we provide a set of best practices for normalization that will allow fair comparison of meta-learning algorithms.

Subfields: Artificial Intelligence, Cancer Research, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cancer Research

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual Normalization: Rethinking Batch Normalization for Online Continual Learning (2022, methodology): “Similar to SN, TaskNorm (Bronskill et al., 2020) combines the moments from BN and IN by a blending factor, which is learned for each task.”
Disability-first Dataset Creation: Lessons from Constructing a Dataset for Teachable Object Recognition with Blind and Low Vision Data Collectors (2021, background): “Achieving this vision, however, requires innovation in few-shot ML techniques [13, 18, 55, 61] that are optimised to work on small amounts of real-world data from people who are”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, background): “However, we note that there also exist works exploring how to apply normalization to meta learning [59], [214], [215], reinforcement learning [216], [217], [218], unsupervised representation learning [219], [220], permutation-equivariant networks [221], [222], graph neural networks [223], ordinary…”
Universal Representation Learning from Multiple Domains for Few-shot Classification (2021, background): “Recent few-shot techniques [5, 13, 29, 44] focus on few-shot learning that generalizes to unseen domains at test time in the recently proposed Meta-Dataset [52].”
A Universal Representation Transformer Layer for Few-Shot Image Classification (2020, unknown): “4 TaskNorm[3] 50.”
Learning a Universal Template for Few-shot Dataset Generalization (2021, methodology): “…> . work is shifting towards a more challenging instance of few-shot classiﬁcation across datasets (Chen et al., 2019; Tseng et al., 2020; Triantaﬁllou et al., 2020; Requeima et al., 2019; Bateni et al., 2020; Bronskill et al., 2020; Saikia et al., 2020; Dvornik et al., 2020; Liu et al., 2021).”