Keeping Neural Networks Simple

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Deep learning in neural networks: An overview (2014, background): “The hope is that the NN will generalize well in later episodes, causing only small errors on previously unseen sequences of input events.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “That is, large absolute weight values are considered unlikely and cost more bits than small ones [91, 294, 135, 97].”
Mollifying Networks (2016, methodology): “This is similar to the linearly decaying probability of layers in Huang et al. (2016a). In our experiments, we use an annealing schedule similar to inverse sigmoid rule in Bengio et al.”
Flat Minima (1997, background): “Hinton and van Camp (1993) and Williams (1994) assume that pushing the posterior weight distribution close to the weight prior leads to \good" generalization (see more details below).”
Simplifying Neural Nets by Discovering Flat Minima (1994, background): “Hinton and van Camp minimize the sum of two terms the rst is conventional error plus variance the other is the distance R p j D log p j D p d between posterior p j D and prior p The problem is to choose a good prior In contrast to their approach our approach does not require a good prior given in ad”