Understanding Deep Features with Computer-Generated Imagery

By Mathieu Aubry, Bryan C. Russell. 2015

153 citations recorded by the source corpus.

We introduce an approach for analyzing the variation of features generated by convolutional neural networks (CNNs) trained on large image datasets with respect to scene factors that occur in natural images. Such factors may include object style, 3D viewpoint, color, and scene lighting configuration. Our approach analyzes CNN feature responses with respect to different scene factors by controlling for them via rendering using a large database of 3D CAD models. The rendered images are presented to a trained CNN and responses for different layers are studied with respect to the input scene factors. We perform a linear decomposition of the responses based on knowledge of the input scene factors and analyze the resulting components. In particular, we quantify their relative importance in the CNN responses and visualize them using principal component analysis. We show qualitative and quantitative results of our study on three trained CNNs: AlexNet [18], Places [43], and Oxford VGG [8]. We observe important differences across the different networks and CNN layers with respect to different scene factors and object categories. Finally, we demonstrate that our analysis based on computer-generated imagery translates to the network representation of natural images.

Subfields: Computer Vision and Pattern Recognition, Environmental Engineering, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Environmental Engineering

In Geology

Citations in the corpus

What makes ImageNet good for transfer learning? (2016, background): “Another line of prior work investigated questions such as how distributed are CNN representations [2, 43], what is the the effect of feature transformations such as binarization [2, 14], how do CNN features change with systematic variations in scene factors [3], how invariant are features to differe”
Making Convolutional Networks Shift-Invariant Again (2019, background): “…manually generated perturbations to the input, such as image transformations (Goodfellow et al., 2009; Lenc & Vedaldi, 2015; Azulay & Weiss, 2019), geometric transforms (Fawzi & Frossard, 2015; Ruderman et al., 2018), and CG renderings with various shape, poses, and colors (Aubry & Russell, 2015).”
Interpreting CNNs via Decision Trees (2018, background): “[1] computed feature distributions of different categories in the CNN feature space.”
Interpretable Convolutional Neural Networks (2017, methodology): “Model diagnosis: Many statistical methods [28, 33, 1] have been proposed to analyze CNN features.”
i-RevNet: Deep Invertible Networks (2018, background): “However, some geometric transformations such as a 3D-rotation seem to have been linearized, as suggested in Aubry & Russell (2015).”
Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis (2015, methodology): “We tune the K value in [1,3,5,7], namely KNN1, KNN3, KNN5 and KNN7 to achieve the best performance.”
Playing for Data: Ground Truth from Computer Games (2016, methodology): “Renderings of object models have been used to analyze the sensitivity of convolutional network features [5].”
Understanding deep convolutional networks (2016, background): “Linearizing Diff(Gj) is equivalent to partially flatten simultaneously all these manifolds, which may explain why manifolds are progressively more regular as the network depth increases [5], but it involves open mathematical questions.”
Marr Revisited: 2D-3D Alignment via Surface Normal Prediction (2016, background): “veloped, such as learning a mapping from CNN features to a 3D light-field embedding space for view-invariant shape retrieval [25] and retrieval using AlexNet [22] pool5 features [2].”
Examining CNN representations with respect to Dataset Bias (2017, methodology): “Visualization of CNNs: In order to open the black box of a CNN, many methods (Zeiler and Fergus 2014; Mahendran and Vedaldi 2015; Simonyan, Vedaldi, and Zisserman 2014; Aubry and Russell 2015; Liu, Shen, and van den Hengel 2015; Dosovitskiy and Brox 2016) have been developed to visualize and analyze”
Interpreting CNN knowledge via an Explanatory Graph (2017, methodology): “Many statistical methods [21, 23, 1] have been proposed to analyze the characteristics of CNN features.”
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks (2016, methodology): “To date, synthetic data have been utilized to generate training data for predicting object pose [24, 17, 9], optical flow [6], semantic segmentation [12, 11, 30, 18], and investigating object features [2, 13].”
Building a Regular Decision Boundary with Deep Networks (2017, background): “Deep networks have been shown to be covariant to such actions [1, 16], to linearize them [24, 22, 30] and combing those strategies permit building invariants to them [3, 20, 9].”
Do Convolutional Neural Networks Learn Class Hierarchy? (2017, background): “This helps in observing visual patterns in the samples, such as possible latent subclasses among the samples of one class (e.g. red and green peppers) [3, 49].”
Mining Object Parts from CNNs via Active Question-Answering (2017, background): “active CNN semanticization: In order to explore the hidden semantics in the CNN, many studies visualized and analyzed patterns of CNN units [44, 23, 33, 1, 21].”
Interactively Transferring CNN Patterns for Part Localization (2017, unknown): “[1] analyzed statistics of CNN features.”