Deep neural networks are biased towards simple functions

73 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning (2019, background): “Speciﬁcally, these challenges are: (1) dependency on large datasets required for bottom-up, data-dependent ML algorithms (Valiant 2000; De Palma, Kiani, and Lloyd 2019), (2) bias in the dataset , enabling the model to emphpoten-tially cause social discrimination and unfair treatment, (3)…”
On the infinite width limit of neural networks with a standard parameterization (2020, background): “The analytic forms for the various kernels inspire some additional interesting observations: • For the NTK parameterization, the kernel resulting from a Bayesian neural network and from gradient descent training of the readout layer of an inﬁnite width network are the same.”
Theoretical Limitations of Self-Attention in Neural Sequence Models (2019, background): “Sensitivity of functions has been studied in computational complexity (Boppana, 1997; Gopalan et al., 2016) and more recently linked to generalization in feedfor-ward networks (De Palma et al., 2018).”
Machine learning and deep learning—A review for ecologists (2022, background): “This would explain why deep neural networks display a bias towards simpler functions (De Palma et al., 2019; Valle-Pérez et al., 2019 ) that increases with the depth of the networks (Huh et al., 2021 ).”