What Happened to My Dog in That Network: Unraveling Top-down Generators in Convolutional Neural Networks

By Patrick W. Gallagher, Shuai Tang, Zhuowen Tu. 2015

6 citations recorded by the source corpus.

Top-down information plays a central role in human perception, but plays relatively little role in many current state-of-the-art deep networks, such as Convolutional Neural Networks (CNNs). This work seeks to explore a path by which top-down information can have a direct impact within current deep networks. We explore this path by learning and using "generators" corresponding to the network internal effects of three types of transformation (each a restriction of a general affine transformation): rotation, scaling, and translation. We demonstrate how these learned generators can be used to transfer top-down information to novel settings, as mediated by the "feature flows" that the transformations (and the associated generators) correspond to inside the network. Specifically, we explore three aspects: 1) using generators as part of a method for synthesizing transformed images --- given a previously unseen image, produce versions of that image corresponding to one or more specified transformations, 2) "zero-shot learning" --- when provided with a feature flow corresponding to the effect of a transformation of unknown amount, leverage learned generators as part of a method by which to perform an accurate categorization of the amount of transformation, even for amounts never observed during training, and 3) (inside-CNN) "data augmentation" --- improve the classification performance of an existing network by using the learned generators to directly provide additional training "inside the CNN".

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Song-Chun Zhu
2. Quanshi Zhang
3. Ying Nian Wu
4. Ruiming Cao
5. Yu Yang
6. Erik Nijkamp
7. Mitch Hill
8. Jianwen Xie

In Computer Vision and Pattern Recognition

1. Song-Chun Zhu
2. Quanshi Zhang
3. Ying Nian Wu
4. Ruiming Cao
5. Alan L. Yuille
6. Erik Nijkamp
7. Luc Van Gool
8. Jianwen Xie

Citations in the corpus

• Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning (2016, background): “Typical research includes end-to-end fine-tuning and transferring CNN knowledge between different categories (Yosinski et al. 2014; Gallagher, Tang, and Tu 2015) and/or datasets (Ganin and Lempitsky 2015).”