Mathematical Foundations for a Compositional Distributional Model of Meaning
By Bob Coecke, Mehrnoosh Sadrzadeh, Stephen Clark. 2010
624 citations recorded by the source corpus.
We propose a mathematical framework for a unification of the distributional theory of meaning in terms of vector space models, and a compositional theory for grammatical types, for which we rely on the algebra of Pregroups, introduced by Lambek. This mathematical framework enables us to compute the meaning of a well-typed sentence from the meanings of its constituents. Concretely, the type reductions of Pregroups are `lifted' to morphisms in a category, a procedure that transforms meanings of constituents into a meaning of the (well-typed) whole. Importantly, meanings of whole sentences live in a single space, independent of the grammatical structure of the sentence. Hence the inner-product can be used to compare meanings of arbitrary sentences, as it is for comparing the meanings of words in the distributional model. The mathematical structure we employ admits a purely diagrammatic calculus which exposes how the information flows between the words in a sentence in order to make up the meaning of the whole sentence. A variation of our `categorical model' which involves constraining the scalars of the vector spaces to the semiring of Booleans results in a Montague-style Boolean-valued semantics.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Natural Language Processing, Computational Linguistics & Semantics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Yoav Goldberg
- Luke S. Zettlemoyer
- Christopher D. Manning
- Jianfeng Gao
- Richard Socher
- Yoshua Bengio
In Computational Theory and Mathematics
- Edward Grefenstette
- Alessandro Lenci
- Phil Blunsom
- Karl Moritz Hermann
- Christopher Potts
- Marco Baroni
- Samuel R. Bowman
- Tamara Polajnar
In Natural Language Processing
- Mohit Bansal
- Phil Blunsom
- Samuel R. Bowman
- Yoav Goldberg
- Christopher D. Manning
- Richard Socher
- Kyunghyun Cho
- Graham Neubig
In Computational Linguistics & Semantics
- Alessandro Lenci
- Phil Blunsom
- Aurélie Herbelot
- Edward Grefenstette
- Jacob Andreas
- Marco Baroni
- Karl Moritz Hermann
- Laura Rimell
Citations in the corpus
- A Convolutional Neural Network for Modelling Sentences (2014, background): “…cases, a composition function is learned and either tied to particular syntactic relations (Guevara, 2010; Zanzotto et al., 2010) or to particular word types (Baroni and Zamparelli, 2010; Coecke et al., 2010; Grefenstette and Sadrzadeh, 2011; Kart-saklis and Sadrzadeh, 2013; Grefenstette, 2013).”
- Deep Unordered Composition Rivals Syntactic Methods for Text Classification (2015, background): “Later, their work was extended to take into account the syntactic relation between words (Erk and Padó, 2008; Baroni and Zamparelli, 2010; Kartsaklis and Sadrzadeh, 2013) and grammars (Coecke et al., 2010; Grefenstette and Sadrzadeh, 2011).”
- Multilingual Distributed Representations without Word Alignment (2013, methodology): “Notably, [3] propose a tensor-based model for semantic composition and, similarly, [4] develop a framework for semantic composition by combining distributional theory with pregroup grammars.”
- Multi-Step Regression Learning for Compositional Distributional Semantics (2013, methodology): “Coecke et al. (2010) have proposed a general formalism for composition in distributional semantics that captures the same notion of function application.”
- Recursive Neural Networks Can Learn Logical Semantics (2014, background): “2012). However, for robust natural language understanding, it is essential to model these phenomena in their full generality on complex linguistic structures. Recent work on the algebraic approach of Clark et al. (2011) has yielded rich frameworks for computing the meanings of complex fragments of n”
- Building a shared world: mapping distributional to model-theoretic semantic spaces (2015, background): “…roughly fall into two groups: a) the fully distributional approaches, which redefine the concepts of formal semantics in distributional terms (Coecke et al., 2011; Bernardi et al., 2013; Grefen-stette, 2013; Hermann et al., 2013; Baroni et al., 2014a; Clarke, 2012); b) the hybrid…”
- An Exploration of Discourse-Based Sentence Spaces for Compositional Distributional Semantics (2015, methodology): “We situate our work within the Categorial framework (Coecke et al., 2010; Baroni et al., 2014; Clark, 2013, 2015) where nouns and sentences are considered atomic types, represented as vectors, and other words as functions, represented as tensors.”
- Measuring Compositionality in Representation Learning (2019, background): “A long line of work in natural language processing (Coecke et al., 2010; Baroni & Zamparelli, 2010; Clark, 2012; Fyshe et al., 2015) focuses on learning composition functions to produce distributed representations of phrases and sentences—that is, for purposes of modeling rather than evaluation.”
- “Not not bad” is not “bad”: A distributional account of negation (2013, methodology): “We borrow from Socher et al. (2012) and others (Baroni and Zamparelli, 2010; Coecke et al., 2010) the idea that the information words refer to is of two sorts: first the semantic content of the word, which can be seen as the sense or reference to the concept the word stands for, and is typically…”
- Evaluating Neural Word Representations in Tensor-Based Compositional Settings (2014, background): “In particular, this is the first large-scale study to date that applies neural word representations in tensor-based compositional distributional models of meaning similar to those formalized by Coecke et al. (2010).”
- Vector Space Models of Lexical Meaning (2015, background): “The bulk of the section will be a presentation of the theoretical framework of Coecke et al. (2010), which has been implemented in Grefenstette et al.”
- Can recursive neural tensor networks learn logical reasoning? (2013, background): “Such formal work could also help to establish what tasks can be accomplished using vector representation models, and what tasks—if any—require more powerful word and phrase representations like the higher order tensors studied in Coecke et al. [21] and Grefenstette [22].”
- Large Language Models (2023, background): “Mathematical approaches to language such as [20, 34, 86, 89] can reveal new structure and provide deeper understanding.”
- A case for deep learning in semantics: Response to Pater (2019, background): “This is not a representational challenge, as Coecke et al. (2011) show with their compositional distributed models of meaning, but rather one arising from the demands of machine learning.”