Diet Networks: Thin Parameters for Fat Genomic
By Adriana Romero, Pierre Luc Carrier, Akram Erraqabi, Tristan Sylvain, Alex Auvolat, Etienne Dejoie, Marc-André Legault, M. Dubé, J. Hussin, Yoshua Bengio. 2016
79 citations recorded by the source corpus.
Learning tasks such as those involving genomic data often poses a serious challenge: the number of input features can be orders of magnitude larger than the number of training examples, making it difficult to avoid overfitting, even when using the known regularization techniques. We focus here on tasks in which the input is a description of the genetic variation specific to a patient, the single nucleotide polymorphisms (SNPs), yielding millions of ternary inputs. Improving the ability of deep learning to handle such datasets could have an important impact in precision medicine, where high-dimensional data regarding a particular patient is used to make predictions of interest. Even though the amount of data for such tasks is increasing, this mismatch between the number of examples and the number of inputs remains a concern. Naive implementations of classifier neural networks involve a huge number of free parameters in their first layer: each input feature is associated with as many parameters as there are hidden units. We propose a novel neural network parametrization which considerably reduces the number of free parameters. It is based on the idea that we can first learn or provide a distributed representation for each input feature (e.g. for each position in the genome where variations are observed), and then learn (with another neural network called the parameter prediction network) how to map a feature's distributed representation to the vector of parameters specific to that feature in the classifier neural network (the weights which link the value of the feature to each of the hidden units). We show experimentally on a population stratification task of interest to medical studies that the proposed approach can significantly reduce both the number of parameters and the error rate of the classifier.
Subfields: Artificial Intelligence, Cancer Research, Genetics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Brett K. Beaulieu-Jones
- Yifan Peng
- Anshul B Kundaje
- Avanti Shrikumar
- Casey S. Greene
- Amr M. Alexandari
- Zhiyong Lu
- Jaewoo Kang
In Cancer Research
In Genetics
- Brett K. Beaulieu-Jones
- Casey S. Greene
- Zhiyong Lu
- Jinbo Xu
- Daniel S. Himmelstein
- Anshul B Kundaje
- Avanti Shrikumar
- Gail L. Rosen
Citations in the corpus
- Application of Artificial Intelligence Technology in Oncology: Towards the Establishment of Precision Medicine (2020, methodology): “The original concept of Diet Networks, with transposed data matrices as inputs to the auxiliary network, can significantly reduce the number of parameters in a fully connected layer [108].”
- Opportunities and obstacles for deep learning in biology and medicine (2017, background): “There are also emerging network architectures, such as Diet Networks for high-dimensional SNP data [371].”
- Mut2Vec: distributed representation of cancerous mutations (2018, methodology): “Another approach involves SNP2Vec [11], which embeds individual SNPs into a continuous space by using a denoising autoencoder [12] and Diet Networks.”
- Fully-Connected Neural Networks with Reduced Parameterization for Predicting Histological Types of Lung Cancer from Somatic Mutations (2020, methodology): “refer to We and Wr as a fat hidden layer and a fat reconstruction layer, respectively [21].”