Do Convolutional Neural Networks Learn Class Hierarchy?

By Bilal Alsallakh, Amin Jourabloo, Mao Ye, Xiaoming Liu, Liu Ren. 2017

232 citations recorded by the source corpus.

Convolutional Neural Networks (CNNs) currently achieve state-of-the-art accuracy in image classification. With a growing number of classes, the accuracy usually drops as the possibilities of confusion increase. Interestingly, the class confusion patterns follow a hierarchical structure over the classes. We present visual-analytics methods to reveal and analyze this hierarchy of similar classes in relation with CNN-internal data. We found that this hierarchy not only dictates the confusion patterns between the classes, it furthermore dictates the learning behavior of CNNs. In particular, the early layers in these networks develop feature detectors that can separate high-level groups of classes quite well, even after a few training epochs. In contrast, the latter layers require substantially more epochs to develop specialized feature detectors that can separate individual classes. We demonstrate how these insights are key to significant improvement in accuracy by designing hierarchy-aware CNNs that accelerate model convergence and alleviate overfitting. We further demonstrate how our methods help in identifying various quality issues in the training data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers (2018, unknown): “To solve this, a visual analytics system called Blocks [47] uses confusion matrices, a technique for summarizing the performance of a classification algorithm, and matrix-level sorting interactions to reveal that class error often occurs in hierarchies.”
Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, unknown): “CNNs for image classification [233], [234].”
CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization (2020, background): “Many visual analytics tools have been developed to help deep learning experts analyze their models and predictions [5,15,23,27,36,37].”