Visual Concepts and Compositional Voting

By Zhishuai Zhang, Cihang Xie, Yuyin Zhou, Vittal Premachandran, Jun Zhu, Lingxi Xie, Alan L. Yuille. 2017

35 citations recorded by the source corpus.

It is very attractive to formulate vision in terms of pattern theory \cite{Mumford2010pattern}, where patterns are defined hierarchically by compositions of elementary building blocks. But applying pattern theory to real world images is currently less successful than discriminative methods such as deep networks. Deep networks, however, are black-boxes which are hard to interpret and can easily be fooled by adding occluding objects. It is natural to wonder whether by better understanding deep networks we can extract building blocks which can be used to develop pattern theoretic models. This motivates us to study the internal representations of a deep network using vehicle images from the PASCAL3D+ dataset. We use clustering algorithms to study the population activities of the features and extract a set of visual concepts which we show are visually tight and correspond to semantic parts of vehicles. To analyze this we annotate these vehicles by their semantic parts to create a new dataset, VehicleSemanticParts, and evaluate visual concepts as unsupervised part detectors. We show that visual concepts perform fairly well but are outperformed by supervised discriminative methods such as Support Vector Machines (SVM). We next give a more detailed analysis of visual concepts and how they relate to semantic parts. Following this, we use the visual concepts as building blocks for a simple pattern theoretical model, which we call compositional voting. In this model several visual concepts combine to detect semantic parts. We show that this approach is significantly better than discriminative methods like SVM and deep networks trained specifically for semantic part detection. Finally, we return to studying occlusion by creating an annotated dataset with occlusion, called VehicleOcclusion, and show that compositional voting outperforms even deep networks when the amount of occlusion becomes large.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, background): “…networks trained to recognize objects are misled by contextual heuristics: they recognize monkeys within their typical environment with high accuracy, yet they label a monkey holding a guitar as a human, presumably because guitars never co-occur with monkeys in the training set (Wang et al., 2018).”
Shortcut learning in deep neural networks (2020, background): “If cows happen to be on grassland for most of the training data, detecting grass instead of cows becomes a successful strategy for solving a classification problem in an unintended way; and indeed many models base their predictions on context [23, 24, 25, 26, 9, 27, 10].”
Mitigating adversarial effects through randomization (2017, neutral): “, 2017), visual concept discovery (Wang et al., 2017) etc.”
Compositional Convolutional Neural Networks: A Deep Architecture With Innate Robustness to Partial Occlusion (2020, background): “Hence, the free energy to be minimized for maximizing the vMF likelihood [25] is: where C is a constant.”
Compositional Convolutional Neural Networks: A Robust and Interpretable Model for Object Recognition Under Occlusion (2020, background): “Hence, the free energy to be minimized for maximizing the vMF likelihood [58] is: where C is a constant.”
Deep Nets: What have They Ever Done for Vision? (2018, background): “d classication challenge before ImageNet). Other Deep Nets architectures also gave enormous performance jumps in other classic tasks like edge detection, semantic segmen3 Figure 3: Figure taken from Wang et al. (2018). Adding occluders cause deep network to fail. Left Panel: The occluding motorbike ”