Building a Regular Decision Boundary with Deep Networks

By Edouard Oyallon. 2017

31 citations recorded by the source corpus.

In this work, we build a generic architecture of Convolutional Neural Networks to discover empirical properties of neural networks. Our first contribution is to introduce a state-of-the-art framework that depends upon few hyper parameters and to study the network when we vary them. It has no max pooling, no biases, only 13 layers, is purely convolutional and yields up to 95.4% and 79.6% accuracy respectively on CIFAR10 and CIFAR100. We show that the nonlinearity of a deep network does not need to be continuous, non expansive or point-wise, to achieve good performance. We show that increasing the width of our network permits being competitive with very deep networks. Our second contribution is an analysis of the contraction and separation properties of this network. Indeed, a 1-nearest neighbor classifier applied on deep features progressively improves with depth, which indicates that the representation is progressively more regular. Besides, we defined and analyzed local support vectors that separate classes locally. All our experiments are reproducible and code will be available online, based on TensorFlow.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Scattering Networks for Hybrid Representation Learning (2018, background): “However, understanding the precise operations performed by those early layers is a complicated [54], [40] and possibly intractable task.”
Probing Representation Forgetting in Supervised and Unsupervised Continual Learning (2022, methodology): “We instead use optimal Linear Probes (LP), commonly used to study unsupervised representations [8] and intermediate layer representations [36, 48], to evaluate CL algorithms and their effectiveness.”
i-RevNet: Deep Invertible Networks (2018, background): “Several works (Oyallon, 2017; Zeiler & Fergus, 2014) observed a phenomenon of progressive separation and contraction in non-invertible networks on limited datasets.”
Greedy Layerwise Learning Can Scale to ImageNet (2018, background): “, 2015) while increasing linear separability (Zeiler & Fergus, 2014; Oyallon, 2017; Jacobsen et al., 2018) of the data.”
Shallow Learning For Deep Networks (2018, background): “, 2015) while increasing linear separability (Zeiler & Fergus, 2014; Oyallon, 2017; Jacobsen et al., 2018) of the data.”