Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

By Sergey Zagoruyko, Nikos Komodakis. 2016

3,142 citations recorded by the source corpus.

Attention plays a critical role in human visual experience. Furthermore, it has recently been demonstrated that attention can also play an important role in the context of applying artificial neural networks to a variety of tasks from fields such as computer vision and NLP. In this work we show that, by properly defining attention for convolutional neural networks, we can actually use this type of information in order to significantly improve the performance of a student CNN network by forcing it to mimic the attention maps of a powerful teacher network. To that end, we propose several novel methods of transferring attention, showing consistent improvement across a variety of datasets and convolutional neural network architectures. Code and models for our experiments are available at this https URL

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, background): “Knowledge of the attention maps (Zagoruyko & Komodakis, 2017; Hu et al., 2018) is also introduced to help the training.”
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, methodology): “While most methods focus on distilling knowledge from a pre-trained teacher model [28, 29, 30, 31, 32], online distillation [33, 34] simultaneously trains multiple models and use their ensemble as the teacher.”
Born Again Neural Networks (2018, methodology): “We construct Wide-ResNets (Zagoruyko & Komodakis, 2016b) and Bottleneck-ResNets (He et al., 2016b) of comparable complexity to their teacher and show that these BAN-as-ResNets surpass their DenseNet teachers.”
Contrastive Representation Distillation (2019, methodology): “Additionally, we also consider other distillation objectives, such as KD discussed in previous section, Attention Transfer Zagoruyko & Komodakis (2016a) and FitNet Romero et al. (2014).”
FEED: Feature-level Ensemble for Knowledge Distillation (2019, methodology): “We chose three types of CNNs to check the applicability of our algorithms on CIFAR-100: ResNet (He et al. 2016), Wide ResNet (Zagoruyko and Komodakis 2016b), and ResNext (Xie et al. 2017).”
Similarity-Preserving Knowledge Distillation (2019, methodology): “Student Teacher Student KD [12] AT [41] SP (ours) Teacher WideResNet-16-1 (0.”
Zero-shot Knowledge Transfer via Adversarial Belief Matching (2019, methodology): “For both datasets we use WideResNet (WRN) architectures (Zagoruyko and Komodakis, 2016b) since they are ubiquitous in the distillation literature and easily allow changing the depth and parameter count.”
Variational Information Distillation for Knowledge Transfer (2019, methodology): “For comparison, we consider the following knowledge transfer methods: the original knowledge distillation (KD) [12], learning without forgetting (LwF) [16], hint based transfer (FitNet) [31], activationbased attention transfer (AT) [31] and polynomial kernelbased neural selectivity transfer (NST) [1”
Paraphrasing Complex Network: Network Compression via Factor Transfer (2018, methodology): “Previous studies such as attention transfer (AT) [30] and knowledge distillation (KD) [10] have achieved meaningful results in the field of knowledge transfer, where their loss function can be collectively summarized as the difference between the attention maps or softened distributions of the teach”
Global-and-local attention networks for visual recognition (2018, methodology): “…between ground-truth ClickMe maps and image-feature importance maps derived from the DCN. Feature importance maps were derived from networks as the L 2 norms of activity from the ﬁnal layer of GALA or SE attention (layer 39, thus generalizing between models with vs. without attention; [43]).”
Network Pruning via Transformable Architecture Search (2019, background): “9 and the temperature T of 4 following [43].”
CBAM: Convolutional Block Attention Module (2018, background): “Applying pooling operations along the channel axis is shown to be effective in highlighting informative regions [34].”
Class-Incremental Learning with Cross-Space Clustering and Controlled Transfer (2022, background): “Keywords: Incremental Learning, Continual Learning, Knowledge Distillation, Transfer Learning.”
Learning with Recoverable Forgetting (2022, positive): “…is calculated as: where λ at is the weight, L ce ( · , · ) is the cross-entropy loss, and L at is the filtered attention distillation loss item [63] that calculates the activated feature-wise similarity of the intermediate features: where F i ∈ R H × W represents the feature F ∈ R H × W × C…”
Class-Incremental Learning for Action Recognition in Videos (2021, positive): “The approaches based on knowledge distillation [14, 31, 44] encourage a model to learn new tasks while mimicking the representations of the old model trained for the previous tasks without their training data.”
Few-Shot Class-Incremental Learning via Relation Knowledge Distillation (2021, positive): “(2) Feature-based knowledge approaches: Feature layer as intermediate representations is used to train the student network (Romero et al. 2014; Zagoruyko and Komodakis 2016; Heo et al. 2019).”
Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation (2021, background): “Parameter efficiency can also be encouraged during training via architecture design [23,29], sparsity inducing priors [33], resource re-allocation [26], or knowledge distillation from a teacher network [34,41].”
PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, background): “Among existing distillation schemes based on intermediate features [17, 70, 56, 16, 78, 29], POD [17] consists in matching global statistics at different feature levels between the old and current models.”