How far can we go without convolution: Improving fully-connected networks

By Zhouhan Lin, Roland Memisevic, Kishore Reddy Konda. 2015

57 citations recorded by the source corpus.

We propose ways to improve the performance of fully connected networks. We found that two approaches in particular have a strong effect on performance: linear bottleneck layers and unsupervised pre-training using autoencoders without hidden unit biases. We show how both approaches can be related to improving gradient flow and reducing sparsity in the network. We show that a fully connected network can yield approximately 70% classification accuracy on the permutation-invariant CIFAR-10 task, which is much higher than the current state-of-the-art. By adding deformations to the training data, the fully connected network achieves 78% accuracy, which is just 10% short of a decent convolutional network.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Variational Information Distillation for Knowledge Transfer (2019, methodology): “Our method yields a student MLP that significantly outperforms the best-reported MLPs [17, 27] in the literature.”
ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training (2021, background): “Several works [37, 40, 59] have questioned if FCNs are also competitive on natural image datasets, such as CIFAR-10 [34].”
Classifying Signals on Irregular Domains via Convolutional Cluster Pooling (2019, methodology): “Indeed, our method outperforms both the best reported fully connected (FC) network [18]”
Do Deep Convolutional Nets Really Need to be Deep and Convolutional? (2016, background): “The only other work we are aware of that achieves comparable high accuracy with non-convolutional MLPs is recent work by Lin et al. (2016).”
Adversarial Reprogramming of Neural Networks (2018, background): “This accuracy is near what is expected from typical fully connected networks (Lin et al., 2015) but with minimal computation cost from the adversary side at inference time.”
The Loss Surface of Deep and Wide Neural Networks (2017, background): “…there are several recent neural network structures which contain a quite wide hidden layer relative to the number of training points e.g. in (Lin et al., 2016) they have 50,000 training samples and the network has one hidden layer with 10,000 hidden units and (Ba & Caruana, 2014) have 1.1…”