Convolutional Deep Belief Networks on CIFAR-10

By Alex Krizhevsky. 2010

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

Citations in the corpus

Semi-supervised Sequence Learning (2015, positive): “While we do not achieve the results attained by sta te of the art convolutional networks, our 2-layer pretrained LM-LSTM is able to exceed the results of the baseline convolutional DBN model [14] despite not using any convolutions and outperfor ms the non pre-trained LSTM.”
Matching Networks for One Shot Learning (2016, methodology): “We ran one-shot experiments on three data sets: two image classification sets (Omniglot [14] and ImageNet [19, ILSVRC-2012]) and one language modeling (Penn Treebank).”
NICE: Non-linear Independent Components Estimation (2014, methodology): “CIFAR-10 is normalized to be in [−1, 1]D. 3We train on unlabeled data for this dataset.”
Representation Learning: A Review and New Perspectives (2012, methodology): “Convolutional versions of the RBM or its extensions have also been developed (Desjardins and Bengio, 2008; Lee et al., 2009a; Taylor et al., 2010) as well as a probabilistic max-pooling operation built into Convolutional Deep Networks (Lee et al., 2009a,b; Krizhevsky, 2010).”
Neural Additive Models: Interpretable Machine Learning with Neural Nets (2020, methodology): “Furthermore, we use ReLU activations capped at n (ReLU-n) [21] to ensure that each ExU unit is active in a small input range, making it easier to model sharp jumps in a function without significantly affecting the global behavior.”
Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model (2016, methodology): “Deep learning is a powerful machine learning technique that has revolutionized image classification (Krizhevsky and Hinton, 2010; Srivastava, et al., 2015) and speech recognition”
Training Deep Neural Networks with 8-bit Floating Point Numbers (2018, methodology): “As a proof of wide-applicability, we additionally trained the CIFAR10-CNN network with the ADAM optimizer [12] and achieved baseline accuracies while using FP 8 GEMMs and FP 16 weight updates.”
Spiking Deep Networks with LIF Neurons (2015, methodology): “Early successes with the MNIST database [1, 2] were expanded to the more difﬁcult but similarly sized CIFAR-10 dataset [3] and Street-view house numbers dataset [4].”
The Importance of Encoding Versus Training with Sparse Coding and Vector Quantization (2011, background): “It has also become popular as the non-linearity in various deep learning architectures (Kavukcuoglu et al., 2010; Nair & Hinton, 2010; Krizhevsky, 2010), and is often referred to as a " shrinkage " function for its role in regularization and sparse coding algorithms (Gregor & LeCun, 2010).”
ReAct: Out-of-distribution Detection With Rectified Activations (2021, methodology): “The idea of rectifying unit activation [26], which is known as Relu6, was used to facilitate the learning of sparse features.”
Learnable Pooling Regions for Image Classification (2013, methodology): “We evaluate our method on the CIFAR-10 and CIFAR-100 datasets [Krizhevsky and Hinton, 2010].”
Better Self-training for Image Classification through Self-supervision (2021, methodology): “We use three benchmark datasets: Street View House Numbers (SVHN) [15], CIFAR-10 [11], and PlantVillage [8].”
DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization (2023, background): “…an anisotropic message passing scheme: where U ℓ , V ℓ , P ℓ , Q ℓ , R ℓ ∈ R d × d are the learnable parameters of layer ℓ , α denotes the ReLU [66] activation, BN denotes the Batch Normalization operator [51], A denotes the aggregation function SUM pooling [116], σ is the sigmoid function, ⊙…”
Learning an Adaptive Learning Rate Schedule (2019, methodology): “In our experiments, we use two data sets: Fashion MNIST [17] and CIFAR-10 [18].”