Deep Sequential Neural Network

By Ludovic Denoyer, Patrick Gallinari. 2014

62 citations recorded by the source corpus.

Neural Networks sequentially build high-level features through their successive layers. We propose here a new neural network model where each layer is associated with a set of candidate mappings. When an input is processed, at each layer, one mapping among these candidates is selected according to a sequential decision process. The resulting model is structured according to a DAG like architecture, so that a path from the root to a leaf node defines a sequence of transformations. Instead of considering global transformations, like in classical multilayer networks, this model allows us for learning a set of local transformations. It is thus able to process data with different characteristics through specific sequences of such local transformations, increasing the expression power of this model w.r.t a classical multilayered network. The learning algorithm is inspired from policy gradient techniques coming from the reinforcement learning domain and is used here instead of the classical back-propagation based gradient descent techniques. Experiments on different datasets show the relevance of this approach.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Adaptive Computation Time for Recurrent Neural Networks (2016, background): “More broadly, learning when to halt can be seen as a form of conditional computing, where parts of the network are selectively enabled and disabled according to a learned policy [3, 6].”
Efficient Content-Based Sparse Attention with Routing Transformers (2020, background): “Conditional computation can also be addressed with reinforcement learning (Denoyer and Gallinari, 2014; Indurthi et al., 2019).”
Large Memory Layers with Product Keys (2019, methodology): “Different methods have been developed like large mixture of experts [39], gating techniques [4, 12, 6] or even reinforcement learning-based approaches [10].”
MaskConnect: Connectivity Learning by Gradient Descent (2018, neutral): “Finally, our approach is also related to conditional computation methods [34,35,36,37,38,39,40,41,42,43], which learn to drop out blocks of units.”
Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning (2017, methodology): “We use REINFORCE (Williams, 1992) to train both the approximation function and tabular representations.”
Conditional Computation in Neural Networks for faster models (2015, neutral): “The Deep Sequential Neural Network (DSNN) model of Denoyer & Gallinari (2014) is possibly closest to our approach.”
BlockDrop: Dynamic Inference Paths in Residual Networks (2017, background): “These gating functions can be learned with reinforcement learning [2, 34, 11].”
Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution (2017, background): “2016); (2) D(2)NNs allow more flexible network architectures and execution sequences including parallel paths, as opposed to architectures with less variance (Denoyer and Gallinari 2014; Shazeer et al. 2017); (3) our D(2)NNs directly optimize arbitrary efficiency metric that is defined by the user, ”
Scaling Vision with Sparse Mixture of Experts (2021, methodology): “In deep learning, the activation of portions of the model can use stochastic neurons [6] or reinforcement learning [4, 17, 53].”
Learning Time/Memory-Efficient Deep Architectures with Budgeted Super Networks (2017, methodology): “We propose to use a policy gradient inspired algorithm as in [5, 1] for learning.”
Deciding How to Decide: Dynamic Routing in Artificial Neural Networks (2017, methodology): “To our knowledge, the family of inference systems we discuss was ﬁrst described by Denoyer & Gallinari (2014).”
You Look Twice: GaterNet for Dynamic Filter Selection in CNNs (2018, background): “Denoyer and Ludovic have proposed a tree structure neural network called Deep Sequential Neural Network [7].”
Changing Model Behavior at Test-Time Using Reinforcement Learning (2017, methodology): “At each metalayer in the network, the controller selects which module will be applied to the activations from the previous metalayer (see also Eigen et al. (2013); Bengio et al. (2015); Andreas et al. (2016); Shazeer et al. (2017); Fernando et al. (2017); Denoyer & Gallinari (2014)).”
Length Adaptive Recurrent Model for Text Classification (2017, methodology): “They usually applied policy-gradient based methods for training, sampling approximation to estimate the intractable expectation of reward [5, 20].”
Anytime Recognition with Routing Convolutional Networks (2019, background): “There are also some recent works [2], [8], [25], [27], [28], [35], [41] which divide a deep network into several building blocks and choose different combinations of these blocks to process speciﬁc samples, in order to accelerate the prediction speed.”