Experimental Support for a Categorical Compositional Distributional Model of Meaning

By Edward Grefenstette, Mehrnoosh Sadrzadeh. 2011

351 citations recorded by the source corpus.

Modelling compositional meaning for sentences using empirical distributional methods has been a challenge for computational linguists. We implement the abstract categorical model of Coecke et al. (2010) using data from the BNC and evaluate it. The implementation is based on unsupervised learning of matrices for relational words and applying them to the vectors of their arguments. The evaluation is based on the word disambiguation task developed by Mitchell and Lapata (2008) for intransitive sentences, and on a similar new experiment designed for transitive sentences. Our model matches the results of its competitors in the first experiment, and betters them in the second. The general improvement in results with increase in syntactic complexity showcases the compositional power of our model.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

A Convolutional Neural Network for Modelling Sentences (2014, background): “…cases, a composition function is learned and either tied to particular syntactic relations (Guevara, 2010; Zanzotto et al., 2010) or to particular word types (Baroni and Zamparelli, 2010; Coecke et al., 2010; Grefenstette and Sadrzadeh, 2011; Kart-saklis and Sadrzadeh, 2013; Grefenstette, 2013).”
A SICK cure for the evaluation of compositional distributional semantic models (2014, methodology): “…DSMs, or CDSMs) have been proposed, with the purpose of representing the meaning of phrases and sentences by composing the distributional representations of the words they contain (Mitchell and Lapata, 2010; Baroni and Zamparelli, 2010; Grefenstette and Sadrzadeh, 2011; Socher et al., 2012).”
Deep Unordered Composition Rivals Syntactic Methods for Text Classification (2015, background): “Later, their work was extended to take into account the syntactic relation between words (Erk and Padó, 2008; Baroni and Zamparelli, 2010; Kartsaklis and Sadrzadeh, 2013) and grammars (Coecke et al., 2010; Grefenstette and Sadrzadeh, 2011).”
SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment (2014, methodology): “…of DSMs (CDSMs) have been proposed, with the purpose of representing the meaning of phrases and sentences by composing the distributional representations of the words they contain (Baroni and Zampar-elli, 2010; Grefenstette and Sadrzadeh, 2011; Mitchell and Lapata, 2010; Socher et al., 2012).”
Illinois-LH: A Denotational and Distributional Approach to Semantics (2014, background): “Following recent work that extends distributional similarities to phrases and sentences (Mitchell and Lapata, 2010; Baroni and Zamparelli, 2010; Grefenstette and Sadrzadeh, 2011; Socher et al., 2012), we deﬁne a phrase vector p to be the pointwise multiplication product of the vectors of the words…”
Multilingual Distributed Representations without Word Alignment (2013, background): “Subsequent work has also attempted to learn distributed semantics of larger structures, allowing us to apply distributed representation to tasks such as sentiment analysis or paraphrase detection (i.a. [1, 3, 12, 14, 21, 25]).”
Multilingual Models for Compositional Distributed Semantics (2014, background): “A CVM learns semantic representations of larger syntactic units given the semantic representations of their constituents (Clark and Pulman, 2007; Mitchell and Lapata, 2008; Baroni and Zamparelli, 2010; Grefenstette and Sadrzadeh, 2011; Socher et al., 2012; Hermann and Blunsom, 2013, inter alia ).”
Multi-Step Regression Learning for Compositional Distributional Semantics (2013, methodology): “We want moreover to test the Regression model against the Categorical model of Grefenstette and Sadrzadeh (2011a) and to design evaluation scenarios allowing a direct comparison with the MV-RNN model of Socher et al. (2012).”
ECNU: One Stone Two Birds: Ensemble of Heterogenous Measures for Semantic Relatedness and Textual Entailment (2014, methodology): “Previous work (Grefenstette and Sadrzadeh, 2011; Socher et al., 2012) performed experiments on their own datasets or on the same datasets which are limited to a few hundred instances of very short sentences with a ﬁxed structure.”
A Comparison of Vector-based Representations for Semantic Composition (2012, result): “In fact, a variety of multiplication-based models can be derived from this framework; and comparisons against component-wise multiplication on phrase similarity tasks yield comparable results (Grefenstette and Sadrzadeh, 2011a; Grefenstette and Sadrzadeh, 2011b).”
Entailment above the word level in distributional semantics (2012, methodology): “One strand attempts to model compo-sitionality with DS methods, representing both primitive and composed linguistic expressions as distributional vectors (Baroni and Zamparelli, 2010; Grefenstette and Sadrzadeh, 2011; Gue-vara, 2010; Mitchell and Lapata, 2010).”
Semantic Compositionality through Recursive Matrix-Vector Spaces (2012, background): “Grefenstette and Sadrzadeh (2011) learn matrices for verbs in a categorical model.”
SICK through the SemEval glasses. Lesson learned from the evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment (2016, methodology): “Recently, several compositional extensions of DSMs (CDSMs) have been proposed, with the purpose of representing the meaning of phrases and sentences by composing the distributional representations of the words they contain (Baroni and Zamparelli 2010; Mitchell and Lapata 2010; Grefenstette and Sadrz”
A Tensor-based Factorization Model of Semantic Compositionality (2013, neutral): “The upper bound is provided by Grefenstette and Sadrzadeh (2011a), based on interannotator agreement.”
Have You Lost the Thread? Discovering Ongoing Conversations in Scattered Dialog Blocks (2017, methodology): “Many models have been proposed (Mitchell and Lapata 2008; Baroni and Zamparelli 2010; Clark et al. 2008; Grefenstette and Sadrzadeh 2011; Zanzotto et al. 2010; Socher et al. 2011, 2012; Kalchbrenner and Blunsom 2013).”