Deriving Neural Architectures from Sequence and Graph Kernels

By Tao Lei, Wengong Jin, Regina Barzilay, Tommi S. Jaakkola. 2017

147 citations recorded by the source corpus.

The design of neural architectures for structured objects is typically guided by experimental insights rather than a formal process. In this work, we appeal to kernels over combinatorial structures, such as sequences and graphs, to derive appropriate neural operations. We introduce a class of deep recurrent neural operations and formally characterize their associated kernel spaces. Our recurrent modules compare the input to virtual reference objects (cf. filters in CNN) via the kernels. Similar to traditional neural operations, these reference objects are parameterized and directly optimized in end-to-end training. We empirically evaluate the proposed class of neural architectures on standard applications such as language modeling and molecular graph regression, achieving state-of-the-art results across these applications.

Subfields: Artificial Intelligence, Materials Chemistry, Model Architectures, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Materials Chemistry

In Model Architectures

In Statistical Learning Theory

Citations in the corpus

Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering (2017, background): “Low-rank tensor inspired neural architectures like in (Lei et al., 2017) could be another choice and we will investigate them in future work.”
Pre-training Graph Neural Networks with Kernels (2018, neutral): “Researchers are already trying to integrate the knowledge acquired from the design of kernels in neural networks [Lei et al. , 2017].”
Hierarchical Graph Representation Learning with Differentiable Pooling (2018, methodology): “network (GNN) models have been proposed in recent years, including methods inspired by convolutional neural networks [5, 8, 11, 16, 21, 24, 29, 36], recurrent neural networks [25], recursive neural networks [1, 30] and loopy belief propagation [7].”
An End-to-End Deep Learning Architecture for Graph Classification (2018, background): “2015; Defferrard, Bresson, and Vandergheynst 2016; Kipf and Welling 2016); and 2) given a set of graphs with different structure and sizes, learn to predict the class labels of unseen graphs (the graph classification problem) (Duvenaud et al. 2015; Atwood and Towsley 2016; Niepert, Ahmed, and Kutzko”
Learning Multimodal Graph-to-Graph Translation for Molecular Optimization (2018, methodology): “Previous work on graph encoders includes convolutional (Scarselli et al., 2009; Bruna et al., 2013; Henaff et al., 2015; Duvenaud et al., 2015; Niepert et al., 2016; Defferrard et al., 2016; Kondor et al., 2018) and recurrent architectures (Li et al., 2015; Dai et al., 2016; Lei et al., 2017).”
Provably Powerful Graph Networks (2019, background): “Lei et al. (2017) showed that their suggested GNN has a theoretical connection to the WL test.”
Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks (2018, background): “…theoretical work on GNNs—with the notable exceptions of Li et al.’s (Li, Han, and Wu 2018) work connecting GNNs to a special form Laplacian smoothing and Lei et al.’s (Lei et al. 2017) work showing that the feature maps generated by GNNs lie in the same Hilbert space as some popular graph kernels.”
Multi-Scale Representation Learning on Proteins (2022, methodology): “Our multi-scale message passing network uses one message passing neural network (MPN) for each layer in the multi-scale graph [Lei et al., 2017, Gilmer et al., 2017].”
Syntax-Directed Variational Autoencoder for Structured Data (2018, background): “The recently developed deep learning models (Duvenaud et al., 2015; Dai et al., 2016; Lei et al., 2017) provide powerful candidates as encoder.”
Adversarial Attack on Graph Structured Data (2018, background): “But recently, people find a fixed number of propagation steps T with various different parameterizations (Li et al., 2015; Dai et al., 2016; Gilmer et al., 2017; Lei et al., 2017) work quite well in various applications.”
A Lightweight Recurrent Network for Sequence Modeling (2019, methodology): “Lei et al. (2018) introduce the simple recurrent unit (SRU), which improves model parallelization by moving matrix computations outside the recurrence.”
Graph Capsule Convolutional Neural Networks (2018, background): “Another popular name for GCNNs is message passing neural networks (MPNNs) (Lei et al., 2017; Gilmer et al., 2017; Dai et al., 2016; Garc´ıa-Dur´an & Niepert, 2017) .”
Glocalized Weisfeiler-Lehman Graph Kernels: Global-Local Feature Maps of Graphs (2017, background): “Secondly, it should be combined with the recent progress in deep learning approaches for graph classification or regression [56]–[59], as our sampling is a form of dropout.”
Training RNNs as Fast as CNNs (2017, methodology): “A theoretical analysis regarding the representational characteristics of (a broader class of) such recurrent architectures is presented in (Lei et al., 2017).”
Simple Recurrent Units for Highly Parallelizable Recurrence (2017, methodology): “n-gram filters) into recurrent computation (Lei et al., 2015; Bradbury et al., 2017; Lei et al., 2017).”
Designing Random Graph Models Using Variational Autoencoders With Applications to Chemical Design (2018, methodology): “, node and edge features) from a different number of hops, or search depth, away from a given node and then map this aggregate information into a continuous latent space, borrowing ideas from graph representation learning [16, 27].”
NeVAE: A Deep Generative Model for Molecular Graphs (2018, background): “Ying, and Leskovec 2017; Lei et al. 2017).”