A High-Throughput Screening Approach to Discovering Good Forms of Biologically Inspired Visual Representation

By Nicolas Pinto, David Doukhan, James J. DiCarlo, David D. Cox. 2009

289 citations recorded by the source corpus.

While many models of biological object recognition share a common set of “broad-stroke” properties, the performance of any one model depends strongly on the choice of parameters in a particular instantiation of that model—e.g., the number of units per layer, the size of pooling kernels, exponents in normalization operations, etc. Since the number of such parameters (explicit or implicit) is typically large and the computational cost of evaluating one particular parameter set is high, the space of possible model instantiations goes largely unexplored. Thus, when a model fails to approach the abilities of biological visual systems, we are left uncertain whether this failure is because we are missing a fundamental idea or because the correct “parts” have not been tuned correctly, assembled at sufficient scale, or provided with enough training. Here, we present a high-throughput approach to the exploration of such parameter sets, leveraging recent advances in stream processing hardware (high-end NVIDIA graphic cards and the PlayStation 3's IBM Cell Processor). In analogy to high-throughput screening approaches in molecular biology and genetics, we explored thousands of potential network architectures and parameter instantiations, screening those that show promising object recognition performance for further analysis. We show that this approach can yield significant, reproducible gains in performance across an array of basic object recognition tasks, consistently outperforming a variety of state-of-the-art purpose-built vision systems from the literature. As the scale of available computational power continues to expand, we argue that this approach has the potential to greatly accelerate progress in both artificial vision and our understanding of the computational underpinning of biological vision.

Subfields: Biophysics, Media Technology, Molecular Biology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Media Technology

In Molecular Biology

In Representation & Self-Supervised Learning

Citations in the corpus

Experience Grounds Language (2020, background): “Playstations (Pinto et al., 2009) and then GPUs (Krizhevsky et al.”
Learning Transferable Architectures for Scalable Image Recognition (2017, methodology): “The proposed method is related to previous work in hyperparameter optimization [43, 4, 5, 53, 54, 6, 39] – especially recent approaches in designing architectures such as Neural Fabrics [47], DiffRNN [40], MetaQNN [3] and DeepArchitect [42].”
Designing Neural Network Architectures using Reinforcement Learning (2016, background): “While there has been some work (Pinto et al., 2009; Bergstra et al., 2013; Domhan et al., 2015) on automated or computer-aided neural network design, new CNN architectures or network design elements are still primarily developed by researchers using new theoretical insights or intuition gained from…”
High-Performance Neural Networks for Visual Object Classification (2011, methodology): “It allows for training large CNNs within days instead of months, such that we can investigate the inﬂuence of various structural parameters by exploring large parameter spaces (Pinto et al. , 2009) and performing error analysis on repeated experiments.”
Emergence of Complex-Like Cells in a Temporal Product Network with Local Receptive Fields (2010, background): “There are several system of this class (Lazebnik et al., 2006; Pinto et al., 2008, 2009; Serre et al., 2007; Yang et al., 2009; Lee et al., 2009; Yu et al., 2009) but speciﬁcally convolutional nets achieve 67% (Jarrett et al., 2009).”
Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning (2016, background): “Note that in this randomly initialized case, we still expect above-chance decoding performance, given past theoretical and empirical work with random networks [20, 36, 42].”
Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity (2016, background): “is suggests that starting from arbitrary weights is unlikely to lead to a good solution. Second, several studies show that the contribution of optimizing the representation layers is relatively small [49,26,44,43,15]. For example, competitive accuracy on CIFAR-10, STL-10, MNIST and MONO datasets can”
Automatic Frankensteining: Creating Complex Ensembles Autonomously (2017, background): “Various studies were able to show that automatic hyperparameter optimization is able to outperform human experts [2, 15].”
Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks (2018, result): “Recent results indicate that more sophisticated and automated approaches can find better hyperparameters – and thus achieve better results – than humans (Pinto et al., 2009; Coates et al., 2011; Bergstra et al., 2011; Snoek et al., 2012; Bardenet et al., 2013; Hutter et al., 2014).”
Teacher Guided Architecture Search (2018, methodology): “Many approaches have been proposed in recent years to automate the discovery of neural network architectures, including random search [32], reinforcement learning [44, 45], evolution [37, 36], and sequential model based optimization (SMBO) [28, 8].”
Learning hyperparameter optimization initializations (2015, methodology): “Recent research proposes automatic hyperparameter optimization techniques that can find good hyperparameters in less time than usual optimization methods and are even able to find better hyperparameters configurations than human experts [2], [3], [4], [5].”
Optimizing Neural Networks that Generate Iimages (2014, background): “The next least programming intensive approach is to do a grid search over some meta-parameter values, as was done quite massively in (Erhan et al., 2009; Pinto et al., 2009).”
PyCUDA and PyOpenCL: A scripting-based approach to GPU run-time code generation (2009, methodology): “…in performance across an array of object and face recognition tasks, consistently outperforming a variety of state-of-the-art purpose-built vision systems from the literature, and that it can oﬀer insight into which computational ideas are most important for achieving this performance [37, 38, 40].”