Grammar Variational Autoencoder

By Matt J. Kusner, Brooks Paige, José Miguel Hernández-Lobato. 2017

956 citations recorded by the source corpus.

Deep generative models have been wildly successful at learning coherent latent representations for continuous data such as video and audio. However, generative modeling of discrete data such as arithmetic expressions and molecular structures still poses significant challenges. Crucially, state-of-the-art methods often produce outputs that are not valid. We make the key observation that frequently, discrete data can be represented as a parse tree from a context-free grammar. We propose a variational autoencoder which encodes and decodes directly to and from these parse trees, ensuring the generated outputs are always valid. Surprisingly, we show that not only does our model more often generate valid outputs, it also learns a more coherent latent space in which nearby points decode to similar discrete outputs. We demonstrate the effectiveness of our learned models by showing their improved performance in Bayesian optimization for symbolic regression and molecular synthesis.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Materials Chemistry, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Materials Chemistry

In Natural Language Processing

In Generative Models

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, methodology): “Recently, MPNNs were extended to allow for continuous updates (Deng et al., 2019; Liu et al., 2019). then optimize for the right structure in the corresponding space, e.g. using VAEs (Kusner et al., 2017; Zhang et al., 2019), or GANs (Yang et al., 2019).”
Variational Autoencoders Pursue PCA Directions (by Accident) (2018, background): “The generative model is pre-dominantely exploited in computer vision [25, 15, 22, 16] with notable exceptions such as generating combinatorial graphs [26].”
Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation (2018, methodology): “JT-VAE has been shown to outperform previous deep generative models for molecule generation, including Character-VAE [9], Grammar-VAE [22], SD-VAE [4] and GraphVAE [39].”
Opportunities and obstacles for deep learning in biology and medicine (2017, background): “Recently, the Grammar Variational Autoencoder, which takes the SMILES grammar into account and is guaranteed to produce syntactically valid SMILES, has been proposed to alleviate this issue [324].”
Learning Multimodal Graph-to-Graph Translation for Molecular Optimization (2018, methodology): “…on molecular optimization approached the graph translation task through generative modeling (G´omez-Bombarelli et al., 2016; Segler et al., 2017; Kusner et al., 2017; Dai et al., 2018; Jin et al., 2018; Samanta et al., 2018; Li et al., 2018a) and reinforcement learning (Guimaraes et al., 2017;…”
Junction Tree Variational Autoencoder for Molecular Graph Generation (2018, methodology): “Given a molecule, we follow the method in Kusner et al. (2017) to construct a grid visualization of its neighborhood.”
GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders (2018, positive): “We used the code and architecture in Kusner et al. (2017) for both baselines, adapting the maximum input length to the smallest possible.”
Learning Deep Generative Models of Graphs (2018, methodology): “We used the code and pretrained models provided by Kusner et al. (2017) to evaluate the performance of CVAE, a VAE-RNN model, and GrammarVAE which improves CVAE by using a decoder that takes into account the SMILES grammar.”
Structured Disentangled Representations (2018, positive): “The latent code z can also serve as a compressed representation for downstream tasks such as text classiﬁcation [Xu et al., 2017], Bayesian optimization [Gómez-Bombarelli et al., 2018, Kusner et al., 2017], and lossy image compression [Theis et al., 2017].”
Tree-to-tree Neural Networks for Program Translation (2018, methodology): “Recently, various neural networks with tree structures have been proposed to employ the structural information of the data [11, 26, 25, 32, 3, 29, 34, 27, 13, 33, 28, 18, 6].”
Syntax-Directed Variational Autoencoder for Structured Data (2018, methodology): “To make a fair comparison, we closely follow the experimental protocols that were set up in Kusner et al. (2017).”
Generating Sentences from Disentangled Syntactic and Semantic Spaces (2019, background): “Variational auto-encoders (VAEs, Kingma and Welling, 2014) are widely used in language generation tasks (Serban et al., 2017; Kusner et al., 2017; Semeniuta et al., 2017; Li et al., 2018b).”
Structure-aware generation of drug-like molecules (2021, methodology): “[8] Matt J Kusner, Brooks Paige, and José Miguel Hernández-Lobato.”
ChemTS: an efficient python library for de novo molecular generation (2017, methodology): “The score of molecule S is described The reward function of ChemTS is deﬁned as ChemTS was compared with two existing methods CVAE [11] and GVAE [12] based on variational autoencoders.”
Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models (2018, methodology): “Some methods try to incorporate SMILES syntax into a network architecture to increase the fraction of valid molecules [49, 50].”