Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
By Micah Goldblum, Jonas Geiping, Avi Schwarzschild, Michael Moeller, Tom Goldstein. 2019
37 citations recorded by the source corpus.
We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike. In this work, we: (1) prove the widespread existence of suboptimal local minima in the loss landscape of neural networks, and we use our theory to find examples; (2) show that small-norm parameters are not optimal for generalization; (3) demonstrate that ResNets do not conform to wide-network theories, such as the neural tangent kernel, and that the interaction between skip connections and batch normalization plays a role; (4) find that rank does not correlate with generalization or robustness in a practical setting.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Ryan Cotterell
- Adina Williams
- Tiago Pimentel
- Yonatan Belinkov
- Naomi Saphra
- Hartmut Bauermeister
- Hannah Dröge
- Michael J. Dinneen
In Computer Vision and Pattern Recognition
- Tiago Pimentel
- Hartmut Bauermeister
- Hannah Dröge
- Ryan Cotterell
- Michael J. Dinneen
- Naomi Saphra
- Cecilia Summers
- Michael Moeller
Citations in the corpus
- Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks (2020, neutral): “A formal description of this process can be found in Algorithm 1, as seen in (Goldblum et al., 2019).”
- Four Things Everyone Should Know to Improve Batch Normalization (2019, background): “As far as we are aware, the effect of regularization on γ and β has not been studied to any great extent — Wu & He (2018) briefly mention weight decay with these parameters, where weight decay was used when training from scratch, but not fine-tuning, two other papers (Goyal et al., 2017; He et al., ”
- Inverting Gradients - How easy is it to break privacy in federated learning? (2020, methodology): “…in [41] which does not contain strides and flattens convolutional features, the dishonest server could set all convolution layers to represent the identity [13], moving the input through the network unchanged up to classification layer, from which the input can be analytically computed as in Prop 2.”
- Pareto Probing: Trading-Off Accuracy and Complexity (2020, result): “, 2019), contrary to the classic bias–variance tradeoff, Goldblum et al. (2020) found that biasing towards small”