Altitude Training: Strong Bounds for Single-Layer Dropout

By Stefan Wager, William Fithian, Sida I. Wang, Percy Liang. 2014

50 citations recorded by the source corpus.

Dropout training, originally designed for deep neural networks, has been successful on high-dimensional single-layer natural language tasks. This paper proposes a theoretical explanation for this phenomenon: we show that, under a generative Poisson topic model with long documents, dropout training improves the exponent in the generalization bound for empirical risk minimization. Dropout achieves this gain much like a marathon runner who practices at altitude: once a classifier learns to perform reasonably well on training examples that have been artificially corrupted by dropout, it will do very well on the uncorrupted test set. We also show that, under similar conditions, dropout preserves the Bayes decision boundary and should therefore induce minimal bias in high dimensions.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

To Drop or Not to Drop: Robustness, Consistency and Differential Privacy Properties of Dropout (2015, background): “In comparison to recent works that analyze dropout for ERM style problems Baldi and Sadowski [2014], Wager et al. [2014], we study the general problem of convex ERM in generalized linear model (GLM) and provide precise generalized error bounds for the same.”
Data Noising as Smoothing in Neural Network Language Models (2017, background): “Additionally, Wager et al. (2014) show that noising can inject appropriate generative assumptions into discriminative models to reduce their generalization error, but do not consider sequence models (Wager et al., 2016).”
Data Augmentation via Levy Processes (2016, methodology): “The analysis of Wager et al. (2014) then builds on a duality between dropout and the above generative model.”
Why Do Better Loss Functions Lead to Less Transferable Features? (2020, background): “Theoretical analysis of dropout has bounded the excess risk of single-layer [70] and two-layer [42] networks.”
The Implicit and Explicit Regularization Effects of Dropout (2020, background): “Dropout has been the focus of several theoretical works studying its properties for both optimization and generalization (Wager et al., 2013, 2014; Baldi & Sadowski, 2013; Helmbold & Long, 2015; Gal & Ghahramani, 2016a; Helmbold & Long, 2017; Cavazza et al., 2017; Mianjy et al., 2018; Mianjy &…”
On the inductive bias of dropout (2014, methodology): “equiring doubling or other parameter-tuning techniques. The impact of dropout on generalization (roughly, how much dropout restricts the search space of the learner) was studied in (Wan et al., 2013; Wager et al., 2014). The latter paper provided improved bounds for learning for a theoretical formul”
Probabilistic Symmetries and Invariant Neural Networks (2020, background): “One body of work has argued that injecting noise into a neural network during training has beneficial effects on training (e.g., Srivastava et al., 2014; Wager et al., 2014).”
Surprising properties of dropout in deep networks (2016, background): “A variety of explanations have been offered (Bachman et al., 2014; Baldi and Sadowski, 2014, 2013; Gal and Ghahramani, 2015; Wager et al., 2014), including the”