Mining Object Parts from CNNs via Active Question-Answering

By Quanshi Zhang, Ruiming Cao, Ying Nian Wu, Song-Chun Zhu. 2017

26 citations recorded by the source corpus.

Given a convolutional neural network (CNN) that is pre-trained for object classification, this paper proposes to use active question-answering to semanticize neural patterns in conv-layers of the CNN and mine part concepts. For each part concept, we mine neural patterns in the pre-trained CNN, which are related to the target part, and use these patterns to construct an And-Or graph (AOG) to represent a four-layer semantic hierarchy of the part. As an interpretable model, the AOG associates different CNN units with different explicit object parts. We use an active human-computer communication to incrementally grow such an AOG on the pre-trained CNN as follows. We allow the computer to actively identify objects, whose neural patterns cannot be explained by the current AOG. Then, the computer asks human about the unexplained objects, and uses the answers to automatically discover certain CNN patterns corresponding to the missing knowledge. We incrementally grow the AOG to encode new knowledge discovered during the active-learning process. In experiments, our method exhibits high learning efficiency. Our method uses about 1/6–1/3 of the part annotations for training, but achieves similar or better part-localization performance than fast-RCNN methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Unsupervised Learning of Neural Networks to Explain Neural Networks (2018, background): “[41, 42] extracted certain neural units to describe an object part in a weakly-supervised manner.”
Interpretable Convolutional Neural Networks (2017, background): “They also disentangled CNN representations in an And-Or graph via active question-answering [37].”
Visual interpretability for deep learning: a survey (2018, methodology): “Up to now, only a few studies have been O rd in ar y C N N s In te rp re ta bl e C N N s Figure 10: Visualization of interpretable filters in the top conv-layer [Zhang et al., 2017c].”
A Competence-aware Curriculum for Visual Concepts Learning via Question Answering (2020, background): “Humans excel at learning visual concepts and their compositions in a questionanswering manner [16,10,18,61,62], which requires a joint understanding of vision and language.”