Do We Need Fully Connected Output Layers in Convolutional Networks?
By Zhongchao Qian, Tyler L. Hayes, Kushal Kafle, Christopher Kanan. 2020
9 citations recorded by the source corpus.
Traditionally, deep convolutional neural networks consist of a series of convolutional and pooling layers followed by one or more fully connected (FC) layers to perform the final classification. While this design has been successful, for datasets with a large number of categories, the fully connected layers often account for a large percentage of the network's parameters. For applications with memory constraints, such as mobile devices and embedded platforms, this is not ideal. Recently, a family of architectures that involve replacing the learned fully connected output layer with a fixed layer has been proposed as a way to achieve better efficiency. In this paper we examine this idea further and demonstrate that fixed classifiers offer no additional benefit compared to simply removing the output layer along with its parameters. We further demonstrate that the typical approach of having a fully connected final output layer is inefficient in terms of parameter count. We are able to achieve comparable performance to a traditionally learned fully connected classification output layer on the ImageNet-1K, CIFAR-100, Stanford Cars-196, and Oxford Flowers-102 datasets, while not having a fully connected output layer at all.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Abu Sebastian
- Luca Benini
- Abbas Rahimi
- Matteo Bruni
- Claudio Baecchi
- Federico Pernici
- Geethan Karunaratne
- Giovanni Cherubini
In Computer Vision and Pattern Recognition
- Matteo Bruni
- Claudio Baecchi
- Federico Pernici
- A. del Bimbo
- Abbas Rahimi
- Abu Sebastian
- Luca Benini
- Geethan Karunaratne
Citations in the corpus
- Constrained Few-shot Class-incremental Learning (2022, methodology): “This cannot be achieved with other methods that replace the fully connected layer with a fixed Hadamard [17] or identity [37] matrix, because they fail to support a larger number of classes than the vector dimensionality of the layer they are connected to.”
- Class-incremental Learning with Pre-allocated Fixed Classifiers (2020, background): “Fixing the final classification layer in multi-task supervised classification has been explored in detail in [24], [47]–[49] showing that it causes little or no reduction in accuracy for common datasets, while allowing a noticeable reduction in trainable parameters.”