Neural Networks: Tricks of the Trade

786 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval (2015, background): “Additional training details : (1) Model parameters are initialized with uniform distribution in the range ( 6 / ( fan in + fan out ) , 6 / ( fan in + fan out )) (Montavon et al., 2012).”
Deep learning in neural networks: An overview (2014, background): “Many additional, older tricks (Sections 5.6.2, 5.6.3) should also be applicable to today’s deep NNs; compare (Montavon et al., 2012; Orr & Müller, 1998).”
Recent advances in convolutional neural networks (2015, methodology): “ard deviation is p 2=n l, where n l = k2 l d l 1, k l is the spatial ﬁlter size in layer land d l 1 is the number of ﬁlters in layer l 1. 2) Stochastic gradient descent: The backpropagation algorithm [46] is the standard training method which uses gradient descent to update the parameters. Standard ”
Methods for interpreting and understanding deep neural networks (2017, background): “However, a good choice of hyperparameters is usually necessary to make them work well on real-world problems, and tricks are often used to make most efficient use of these methods and extend their capabilities [10,25,47].”
Cyclical Learning Rates for Training Neural Networks (2015, neutral): “It is well known that too small a learning rate will make a training algorithm converge slowly while too large a learning rate will make the training algorithm diverge [2].”