RNNs Implicitly Implement Tensor Product Representations

By R. Thomas McCoy, Tal Linzen, Ewan Dunbar, Paul Smolensky. 2018

61 citations recorded by the source corpus.

Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies). Such regularities motivate our hypothesis that RNNs that show such regularities implicitly compile symbolic structures into tensor product representations (TPRs; Smolensky, 1990), which additively combine tensor products of vectors representing roles (e.g., sequence positions) and vectors representing fillers (e.g., particular words). To test this hypothesis, we introduce Tensor Product Decomposition Networks (TPDNs), which use TPRs to approximate existing vector representations. We demonstrate using synthetic data that TPDNs can successfully approximate linear and tree-based RNN autoencoder representations, suggesting that these representations exhibit interpretable compositional structure; we explore the settings that lead RNNs to induce such structure-sensitive representations. By contrast, further TPDN experiments show that the representations of four models trained to encode naturally-occurring sentences can be largely approximated with a bag of words, with only marginal improvements from more sophisticated structures. We conclude that TPDNs provide a powerful method for interpreting vector representations, and that standard RNNs can induce compositional sequence representations that are remarkably well approximated by TPRs; at the same time, existing training tasks for sentence representation learning may not be sufficient for inducing robust structural representations.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference (2019, methodology): “Thus, the results of McCoy et al. (2019) also suggest that the training set rather than the architecture might be to blame for poor compositional understanding.”
What Does BERT Learn about the Structure of Language? (2019, methodology): “Following McCoy et al. (2019), we train our TPDN model on the premise sentences in the SNLI corpus (Bowman et al., 2015).”
Learning Distributed Representations of Symbolic Structure Using Binding and Unbinding Operations (2018, background): “Analyses [19] also show that standard RNNs can be interpreted as learning TPRs in their hidden representations.”
Discovering the Compositional Structure of Vector Representations with Role Learning Networks (2019, methodology): “ROLE is an extension of the Tensor-Product Encoder (TPE) introduced in McCoy et al. [12] , which produces a linearly-transformed TPR given a string of symbols and pre-assigned role labels for each symbol (see Appendix A.1 for details).”