Syntax-Directed Variational Autoencoder for Structured Data

By Hanjun Dai, Yingtao Tian, Bo Dai, Steven S. Skiena, Le Song. 2018

358 citations recorded by the source corpus.

Deep generative models have been enjoying success in modeling continuous data. However it remains challenging to capture the representations for discrete structures with formal grammars and semantics, e.g., computer programs and molecular structures. How to generate both syntactically and semantically correct data still remains largely an open problem. Inspired by the theory of compiler where the syntax and semantics check is done via syntax-directed translation (SDT), we propose a novel syntax-directed variational autoencoder (SD-VAE) by introducing stochastic lazy attributes. This approach converts the offline SDT check into on-the-fly generated guidance for constraining the decoder. Comparing to the state-of-the-art methods, our approach enforces constraints on the output space so that the output will be not only syntactically valid, but also semantically reasonable. We evaluate the proposed model with applications in programming language and molecules, including reconstruction and program/molecule optimization. The results demonstrate the effectiveness in incorporating syntactic and semantic constraints in discrete generative models, which is significantly better than current state-of-the-art approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Model Architectures

Citations in the corpus

Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation (2018, methodology): “JT-VAE has been shown to outperform previous deep generative models for molecule generation, including Character-VAE [9], Grammar-VAE [22], SD-VAE [4] and GraphVAE [39].”
Learning Multimodal Graph-to-Graph Translation for Molecular Optimization (2018, methodology): “…approached the graph translation task through generative modeling (G´omez-Bombarelli et al., 2016; Segler et al., 2017; Kusner et al., 2017; Dai et al., 2018; Jin et al., 2018; Samanta et al., 2018; Li et al., 2018a) and reinforcement learning (Guimaraes et al., 2017; Olivecrona et al.,…”
Junction Tree Variational Autoencoder for Molecular Graph Generation (2018, methodology): “Compared to the ﬁgure in Dai et al. (2018), our neighborhood does not contain molecules with huge rings (with more than 7 atoms), which rarely occur in the dataset.”
GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models (2018, background): “Most recently, SD-VAE (Dai et al., 2018) introduced a grammar-based approach to generate structured data, including molecules and parse trees.”
A Comprehensive Survey on Graph Neural Networks (2019, background): “We discuss the theoretical foundation of graph neural networks from different perspectives.”
Generating Sentences from Disentangled Syntactic and Semantic Spaces (2019, background): “A line of studies (Kusner et al., 2017; G´omez-Bombarelli et al., 2018; Dai et al., 2018) propose to impose context-free grammars (CFGs) as hard constraints in the VAE decoder, so that they could generate syntactically valid outputs of programs, molecules, etc.”
Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation (2020, methodology): “Grammar based methods have been used for a variety of tasks such as program translation [9], conditional program generation [71,72], grammar induction [32] and generative modelling on structures with syntax [37,12], such as molecules.”
Mol-CycleGAN: a generative model for molecular optimization (2019, methodology): “Junction Tree Variational Autoencoder JT-VAE [30] (Junction Tree Variational Autoencoder) is a method based on VAE, which works on graph structures of compounds, in contrast to previous methods which utilize the SMILES representation of molecules [19, 20, 27].”
MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design (2022, methodology): “There are two common [5], SD-VAE [124] GrammarVAE [125]) learn a VAE-based model which encodes and decodes SMILES strings and sample new molecules (strings) via decoding from a Gaussian prior.”
Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation (2019, positive): “ representation of GrammarVAE as one of our baselines. 2 Further improvements over the GrammarVAE came from introducing syntax-directed VAEs which use attribute grammars to introduce semantic meaning [28]. Junction-Tree VAEs introduce supernodes to transform molecular graphs into trees [29]. These m”
Inverse molecular design using machine learning: Generative models for matter engineering (2018, methodology): “Initially, VAEs were proposed for encoding characters of SMILES and then extended to take into account grammar and syntax features, which improve the generation of syntactically valid structures ( 70 , 71 ).”
Efficient Graph Generation with Graph Recurrent Attention Networks (2019, methodology): “For auto-regressive models, Dai et al. [5] chose to model the sequential graphs using RNNs, and enforced constraints in the output space using attribute grammars to ensure the syntactic and semantic validity.”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, background): “Another line of works [195, 196] introduce the notion of program graphs [197] to explicitly model the structures underlying PL code to assist generation.”
NeVAE: A Deep Generative Model for Molecular Graphs (2018, methodology): “…generated by several state of the art competing methods: (i) GraphVAE (Simonovsky and Komodakis 2018), (ii) GrammarVAE (Kusner et al. 2017), (iii) CVAE (Gómez-Bombarelli et al. 2016), (iv) SDVAE (Dai et al. 2018), (v) JTVAE (Jin, Barzilay, and Jaakkola 2018) and (vi) CGVAE (Liu et al. 2018).”
Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining (2020, methodology): “This is identical to the scoring function from references [29, 5, 21, 65, 64], except that we bound the score below by − 6 to prevent points with highly-negative scores from substantially impacting the optimization procedure.”