Deep Networks with Internal Selective Attention through Feedback Connections

By Marijn F. Stollenga, Jonathan Masci, Faustino J. Gomez, J. Urgen Schmidhuber. 2014

265 citations recorded by the source corpus.

Traditional convolutional neural networks (CNN) are stationary and feedforward. They neither change their parameters during evaluation nor use feedback from higher to lower layers. Real brains, however, do. So does our Deep Attention Selective Network (dasNet) architecture. DasNets feedback structure can dynamically alter its convolutional filter sensitivities during classification. It harnesses the power of sequential processing to improve classification performance, by allowing the network to iteratively focus its internal attention on some of its convolutional filters. Feedback is trained through direct policy search in a huge million-dimensional parameter space, through scalable natural evolution strategies (SNES). On the CIFAR-10 and CIFAR-100 datasets, dasNet outperforms the previous state-of-the-art model.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Spatial Transformer Networks (2015, methodology): “[26] use a policy based on a network’s activations to gate the responses of the network’s filters for a subsequent forward pass of the same image and so can allow attention to specific features.”
Squeeze-and-Excitation Networks (2017, methodology): “a softmax or sigmoid) and sequential techniques [12, 41].”
Ask Me Anything: Dynamic Memory Networks for Natural Language Processing (2015, background): “Attention mechanisms are generally useful and can improve image classification [19], automatic image captioning [20] and machine translation [21, 22].”
Striving for Simplicity: The All Convolutional Net (2014, methodology): “…more complex activation functions (Goodfellow et al., 2013; Lin et al., 2014; Srivastava et al., 2013) techniques for improving class inference (Stollenga et al., 2014; Srivastava & Salakhutdinov, 2013) as well as procedures for improved regularization (Zeiler & Fergus, 2013; Springenberg &…”
Dynamic Memory Networks for Visual and Textual Question Answering (2016, background): “They can beneﬁt image classiﬁcation (Stollenga et al., 2014), generating captions for images (Xu et al., 2015), among others mentioned below, and machine translation (Cho et al., 2014; Bahdanau et al., 2015; Luong et al., 2015).”
Gated Feedback Recurrent Neural Networks (2015, methodology): “…vector h t which is a function of all the previous symbols x 1 , · · · , x t − 1 and current symbol x t : This approach of using a neural network to model a probability distribution over sequences is widely used, for instance, in language modeling (see, e.g., Bengio et al., 2001; Mikolov, 2012).”
Towards Deep Neural Network Architectures Robust to Adversarial Examples (2014, background): “Since modeling such recursions in DNNs is notoriously hard and often relies on additional techniques such as reinforcement learning Stollenga et al. (2014); Mnih et al. (2014), we will at ﬁrst investigate explicit inclusion of input generalization as an additional objective for the standard DNN…”
Global-and-local attention networks for visual recognition (2018, background): “While biology is sometimes mentioned as a source of inspiration [1–7], the attentional mechanisms that have been considered by the computer vision community remain limited in comparison to the richness and diversity of the processes used by our visual system (see [8] for a review).”
Residual Attention Network for Image Classification (2017, background): “However, a new process, reinforcement learning [30] or optimization [2] is involved during the training step.”
SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning (2016, background): “In this way, visual attention can be considered as a dynamic feature extraction mechanism that combines contextual fixations over time [19, 26].”
Dual Attention Networks for Multimodal Reasoning and Matching (2016, methodology): “A number of methods have recently adopted visual attention to benefit image classification [21, 28], image generation [6], image captioning [33], visual question answering [35, 26, 32], etc.”
Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks (2017, background): “[61] designed a deep attention selective network for image classification.”
Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning (2017, background): “Routing networks share a common goal with techniques for automated selective transfer learning using attention (Rajendran et al., 2017) and learning gating mechanisms between representations (Stollenga et al., 2014), (Misra et al., 2016), (Ruder et al., 2017).”
Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree (2015, neutral): “dasNet [37] - 9.”
Conditional Computation in Neural Networks for faster models (2015, background): “Stollenga et al. (2014) recently proposed to learn a sequential decision process over the ﬁlters of a convolutional neural network (CNN).”
Large-Margin Softmax Loss for Convolutional Neural Networks (2016, neutral): “…Rate FitNet (Romero et al., 2015) 35.04 NiN (Lin et al., 2014) 35.68 Maxout (Goodfellow et al., 2013) 38.57 DSN (Lee et al., 2015) 34.57 dasNet (Stollenga et al., 2014) 33.78 All-CNN (Springenberg et al., 2015) 33.71 R-CNN (Liang & Hu, 2015) 31.75 GenPool (Lee et al., 2016) 32.37 accuracy over…”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, background): “Mixtures of experts models, or more generally soft parameter sharing models, have been considered in the multi-task and lifelong learning setting [Stollenga et al., 2014, Aljundi et al., 2017, Misra et al., 2016, Ruder et al., 2017, Rajendran et al., 2017], but do not allow for nearly the level of…”
Ray Interference: a Source of Plateaus in Deep Reinforcement Learning (2019, background): “Assuming knowledge of the components of the objective, the multi-task literature oﬀers a plethora of approaches to avoid negative interference, from modular or multi-head architectures to gating or attention mechanisms [3, 7, 32, 37, 41, 44].”