Neural Networks: Tricks of the Trade

By J. Hartmanis, Takeo Kanade. 2002

1,932 citations recorded by the source corpus.

Subfields: Artificial Intelligence, General Social Sciences, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Optimization & Training

Citations in the corpus

Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval (2015, background): “Additional training details: (1) Model parameters are initialized with uniform distribution in the range (−√6/(fanin + fanout),√6/(fanin + fanout)) (Montavon et al., 2012).”
Don't Decay the Learning Rate, Increase the Batch Size (2017, background): “Supporting this view, we note that most researchers employ early stopping (Prechelt, 1998), whereby we intentionally prevent the network from reaching a minimum.”
Group Normalization (2018, methodology): “To normalize hidden features, initialization methods (LeCun et al. 1998; Glorot and Bengio 2010; He et al. 2015) have been derived based on strong assumptions of feature distributions, which can become invalid when training evolves.”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “Therefore tanh has been preferred over logistic functions (LeCun et al., 1991; 1998).”
Representation Learning: A Review and New Perspectives (2012, positive): “This has motivated an earlier book on the “Tricks of the Trade” (Orr and Muller, 1998) of which LeCun et al. (1998a) is still relevant for training deep architectures, in particular what concerns initialization, ill-conditioning and stochastic gradient descent.”
Data-dependent Initializations of Convolutional Neural Networks (2015, positive): “Early approaches to data-driven initializations showed that whitening the activations at all layers can mitigate the vanishing gradient problem (LeCun et al., 1998), but it does not ensure all layers train at an equal rate.”
Train longer, generalize better: closing the generalization gap in large batch training of neural networks (2017, background): “Keskar et al. (2017) focused on a long observed phenomena (LeCun et al., 1998a) – that when a large batch size is used while training DNNs, the trained models appear to generalize less well.”
Deep learning in neural networks: An overview (2014, background): “Many additional, older tricks (Sections 5.6.2, 5.6.3) should also be applicable to today’s deep NNs; compare (Montavon et al., 2012; Orr & Müller, 1998).”
Recent advances in convolutional neural networks (2015, methodology): “2) Stochastic gradient descent: The backpropagation algo-rithm [58] is the standard training method which uses gradient descent to update the parameters.”
Methods for interpreting and understanding deep neural networks (2017, background): “However, a good choice of hyperparameters is usually necessary to make them work well on real-world problems, and tricks are often used to make most eﬃcient use of these methods and extend their capabilities [8, 23, 35].”
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation (2015, background): “[6, 16]), we will consider general multilayered network structures with arbitrary continuous neurons and pooling functions based on generalized p-means.”
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, methodology): “The poor scaling of naïve methods like grid search with dimensionality has driven interest in more sophisticated hyperparameter optimization methods over the past years [2, 3, 4, 5, 6, 7, 8, 9, 10, 11].”
Deep Learning in Finance (2016, methodology): “For ultra-large data-sets, we use mini-batches and stochastic gradient descent (SGD) to perform this optimization, see LeCun et al. (2012).”
Quantum-chemical insights from deep tensor neural networks (2016, methodology): “…equilibrium structures within the GDB-9 database and MD trajectories of selected molecules of chemical relevance demonstrates the feasibility of developing a universal machine learning model that can capture both compositional and conﬁgurational degrees of freedom in the vast chemical space.”
Machine learning of molecular electronic properties in chemical compound space (2013, background): “For more details on training neural networks, the reader is referred to [65, 66]”
A Simple Way to Initialize Recurrent Networks of Rectified Linear Units (2015, methodology): “Further developments of the HF approach look promising [35, 25] but are much harder to imple ent than popular simple methods such as stochastic gradient descent with momentum [34] or ad aptive learning rates for each weight that depend on the history of its gradients [5, 14].”