Tiled convolutional neural networks

By Quoc V. Le, Jiquan Ngiam, Zhenghao Chen, Daniel Jin hao Chia, Pang Wei Koh, Andrew Y. Ng. 2010

352 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Deeply-Supervised Nets (2014, background): “[12], or a hybrid form [18]. Signiﬁcant performance gain has been observed, especially in the presence of large amount of training data, when deep learning techniques are used for image classiﬁcation [11, 16] and speech recognition [4]. On the one hand, hierarchical and recursive networks [7, 10, 12”
Visualizing and Understanding Convolutional Networks (2013, background): “Motivated by the latter’s short-coming, [19] (extending an idea by [2]) show how the Hessian of a given unit may be computed numerically around the optimal response, giving some insight into invariances.”
Spectral Networks and Locally Connected Networks on Graphs (2013, methodology): “s. Using arbitrary ﬁlters instead of generic fully connected layers reduces the complexity to O(n) parameters per feature map, as does using the metric structure by building a “locally connected” net [8, 17]. Using the two together gives O(kS) parameters, where kis the number of feature maps and Sis”
Deep Convolutional Networks on Graph-Structured Data (2015, background): “There have been several works which have explored architectures using the so-called local receptive ﬁelds [6, 4, 14], mostly with applications to image recognition.”
Representation Learning: A Review and New Perspectives (2012, methodology): “Similarly, Le et al. (2011b) recently showed that adding a regularization term of the form ∑ t ∑ j s3(Wjx (t)) to a linear auto-encoder with tied weights, where s3 is a nonlinear convex function, yields an efficient algorithm for learning linear ICA.”
Building high-level features using large scale unsupervised learning (2011, methodology): “Our algorithm is built upon these ideas, especially [15].”
Recent advances in convolutional neural networks (2015, methodology): “size to control the distance over which weights are shared. When the tile size k is 1, the units within each map will have the same weights, and tiled CNN becomes identical to the traditional CNN. In [25], their experiments on the NORB and CIFAR-10 datasets show that k = 2 achieves the best results.”
Disentangling Factors of Variation via Generative Entangling (2012, background): “nen, 1996). Since then, the same basic strategy has reappeared in a number of dierent models and learning paradigms, including topological independent component analysis (Hyvarinen and Hoyer, 2000; Le et al., 2010), invariant predictive sparse decomposition (IPSD) (Kavukcuoglu et al., 2009), as well”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “n to be understood. There are also many architectural choices that are relevant in the case of convolutional architectures (e.g. for modeling images, time-series or sound) (LeCun et al., 1989, 1998b; Le et al., 2010) in which hidden units have local receptive ﬁelds. Their discussion is postponed to ”
Exploiting Cyclic Symmetry in Convolutional Neural Networks (2016, background): “The most prominent example of this is the standard architecture of convolutional neural networks (CNNs), which typically consist of alternating convolutional layers and pooling layers.”
Deep Learning using Linear Support Vector Machines (2013, methodology): “ invariant hidden latent representations. The corresponding hidden variables of data samples are then treated as input and fed into linear (or kernel) SVMs (Huang & LeCun, 2006; Lee et al., 2009; Quoc et al., 2010; Coates et al., 2011). This technique usually improves performance but the drawbac”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “ts from the other processors to compute its result, as the operation sums over Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis 1:21 (a) Locally Connected Networks[180] (b) TreeNets[148] Fig. 16. Model Parallelism Schemes all features. To mitigate this problem, L”
Locally Scale-Invariant Convolutional Neural Networks (2014, background): “Tiled convolutional neural networks [8] learn invariances implicitly by square-root pooling hidden units that are computed by partially un-tied weights.”
Rotation Equivariant Vector Field Networks (2016, background): “[21] pool across several untied filters, thus letting the network learn the type of invariance.”
Feature learning and deep architectures: new directions for music informatics (2013, background): “One such instance is the use of local receptive fields in deep learning, common in CNNs and, more recently, tiled networks [33].”
Deep Learning Identity-Preserving Face Space (2013, background): “Then, a matrix V (1), where Vij ∈ {0, 1} encodes the 2D topography of the pooling layer [13], downsamples each of these feature map to 48 × 48 in order to reduce the number of parameters need to be learned and obtain more robust features.”