Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems

By Geoffrey E. Hinton. 1991

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Experience Grounds Language (2020, background): “The tension of modeling symbols and distributed representations was nicely articulated by Smolensky (1990), and alternative representations (Kohonen, 1984; Hinton et al., 1986; Barlow, 1989) and approaches to structure and composition (Erk and Padó, 2008; Socher et al., 2012) span decades of…”
DeBERTa: Decoding-enhanced BERT with Disentangled Attention (2020, methodology): “(1)In this sense, our model shares some similarity to Tensor Product Representation [23, 24, 25] where a word is represented using a tensor product of its filler (content) vector and its role (position) vector.”
Neural Turing Machines (2014, background): “In response to this criticism, neural network researchers including Hinton (Hinton, 1986), Smolensky (Smolensky, 1990), Touretzky (Touretzky, 1990), Pollack (Pollack, 1990), Plate (Plate, 2003), and Kanerva (Kanerva, 2009) investigated specific mechanisms that could support both variable-binding and”
Zero-Shot Text-to-Image Generation (2021, methodology): “Prompts like the latter require the model to perform variable binding (Smolensky, 1990; Greff et al., 2020) – it is the hedgehog that is in the christmas sweater, not the dog.”
On the Binding Problem in Artificial Neural Networks (2020, methodology): “…has a long history in connectionism that goes back to at least Fodor and Pylyshyn (1988); Marcus (2003) and has been repeatedly emphasized (e.g. Smolensky, 1990; Bader and Hitzler, 2005), especially in recent years (e.g. Lake et al., 2017; Battaglia et al., 2018; Hamrick, 2019; Garnelo and…”
Relational inductive biases, deep learning, and graph networks (2018, background): “…objects were developed in domains such as analogy-making, linguistic analysis, symbol manipulation, and other forms of relational reasoning (Smolensky, 1990; Hinton, 1990; Pollack, 1990; Elman, 1991; Plate, 1995; Eliasmith, 2013), as well as more integrative theories for how the mind works…”
Deep Learning: Methods and Applications (2014, background): “, they can potentially perform powerful reasoning over structures, as argued in various historical publications as well as recent essays [38, 156, 286, 288, 292, 336, 335].”
A semantic matching energy function for learning with multi-relational data (2013, methodology): “More precisely, it is connected to models aiming at composing such vector embeddings for obtaining phrase or context representations (Mitchell and Lapata 2008), via tensor products (Smolensky 1990) or matrix operations (Paccanaro and Hinton 2001; Socher et al.”
From machine learning to machine reasoning (2011, methodology): “Although there are connectionist approaches to variable binding (e.g., Smolensky, 1990), they cannot avoid the computational complexity of first-order logic problems.”
Experimental Support for a Categorical Compositional Distributional Model of Meaning (2011, background): “§4 for definition) can take word-order into account (Smolensky, 1990) or even some more complex syntactic relations, as described in Clark and Pulman (2007).”
A Structured Vector Space Model for Word Meaning in Context (2008, methodology): “There have been several approaches in the literature (Smolensky, 1990; Sch ¨ utze, 1998; Kintsch, 2001; McDonald and Brew, 2004; Mitchell and La-pata, 2008) that compute meaning in context from lemma vectors.”
A Review of Relational Machine Learning for Knowledge Graphs (2015, background): “For a further discussion of the tensor product to create composite latent representations, see [88]–[90].”
A Survey of Inductive Biases for Factorial Representation-Learning (2016, background): “A faithful representation preserves the information in the observation [49], with as little distortion as possible.”
Learning to Relate Images (2011, methodology): “Like an auto-encoder, this model can be trained with gradient based optimization.”
Holographic Embeddings of Knowledge Graphs (2015, background): “(9) as holographic embeddings (H OL E) of KGs.”
Domain and Function: A Dual-Space Model of Semantic Relations and Compositions (2012, methodology): “Mono space was created in order to test the hypothesis that it is useful to separate the domain and function spaces; mono space serves as a baseline.”