Distributional Memory: A General Framework for Corpus-Based Semantics

By Marco Baroni, Alessandro Lenci. 2010

684 citations recorded by the source corpus.

Research into corpus-based semantics has focused on the development of ad hoc models that treat single tasks, or sets of closely related tasks, as unrelated challenges to be tackled by extracting different kinds of distributional information from the corpus. As an alternative to this “one task, one model” approach, the Distributional Memory framework extracts distributional information once and for all from the corpus, in the form of a set of weighted word-link-word tuples arranged into a third-order tensor. Different matrices are then generated from the tensor, and their rows and columns constitute natural spaces to deal with different semantic problems. In this way, the same distributional information can be shared across tasks such as modeling word similarity judgments, discovering synonyms, concept categorization, predicting selectional preferences of verbs, solving analogy problems, classifying relations between word pairs, harvesting qualia structures with patterns or example pairs, predicting the typical properties of concepts, and classifying verbs into alternation classes. Extensive empirical testing in all these domains shows that a Distributional Memory implementation performs competitively against task-specific algorithms recently reported in the literature for the same tasks, and against our implementations of several state-of-the-art methods. The Distributional Memory approach is thus shown to be tenable despite the constraints imposed by its multi-purpose nature.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank (2013, background): “Often, co-occurrence statistics of a word and its context are used to describe each word (Turney and Pantel, 2010; Baroni and Lenci, 2010), such as tf-idf.”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, background): “Often, co-occurrence statistics of a word and its context are used to describe each word (Turney and Pantel, 2010; Baroni and Lenci, 2010), such as tf-idf.”
Zero-Shot Learning Through Cross-Modal Transfer (2013, background): “In these approaches, words are represented as vectors of distributional characteristics – most often their co-occurrences with words in context [26, 9, 1, 32].”
SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation (2014, background): “…that can require rich, structured conceptual knowledge to compute accurately, similarity estimation constitutes an effective proxy evaluation for general-purpose representation-learning models whose ultimate application is variable or unknown (Collobert and We-ston, 2008; Baroni and Lenci, 2010).”
Advances in Pre-Training Distributed Word Representations (2017, background): “Their impressive capability at transfering to new problems suggests that they are capturing important statistics about the training corpora (Baroni and Lenci, 2010).”
Improving Distributional Similarity with Lessons Learned from Word Embeddings (2015, background): “While other definitions of contexts have been studied (Padó and Lapata, 2007; Baroni and Lenci, 2010; Levy and Goldberg, 2014a) this work focuses on fixed-window bag-of-words contexts.”
Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors (2014, methodology): “Finally, the Battig (battig) test set introduced by Baroni et al. (2010) includes 83 concepts from 10 categories.”
Multimodal Distributional Semantics (2013, methodology): “The text-based distributional vector representing a word, taken there from a state-of-the-art distributional semantic model (Baroni & Lenci, 2010), is concatenated with a vector representing the same word with visual features , extracted from all the images in the ESP-Game collection we also use…”
Dependency-Based Word Embeddings (2014, background): “Speciﬁcally, following work in sparse vector-space models (Lin, 1998; Pad´o and Lapata, 2007; Baroni and Lenci, 2010), we experiment with syntactic contexts that are derived from automatically produced dependency parse-trees.”
Jointly optimizing word representations for lexical and sentential tasks with the C-PHRASE model (2015, background): “This is reminiscent of how the parameters of some compositional distributional seman-1 tic models are estimated by optimizing the prediction of the contexts in which phrases occur in corpora (Baroni and Zamparelli, 2010; Guevara, 2010; Dinu et al., 2013).”
All-but-the-Top: Simple and Effective Postprocessing for Word Representations (2017, background): “…(Almuhareb, 2006) contains 402 concepts which fall into 21 categories; the ESSLLI 2008 Distributional Semantic Workshop shared-task dataset (Baroni et al., 2008) that contains 44 concepts in 6 categories; and the Battig test set (Baroni and Lenci, 2010) that contains 83 words in 10 categories.”
Distributional vectors encode referential attributes (2015, background): “…in a vector space is an effective strategy to address the many linguistic phenomena that are better characterized in gradient rather than discrete terms, such as synonymy, selectional preferences, and semantic priming (Baroni and Lenci, 2010; Erk et al., 2010; Pad´o and Lapata, 2007, among others).”
Formal Distributional Semantics: Introduction to the Special Issue (2016, background): “Distributional models have been shown to reliably correlate with human judgments on semantic similarity, as well as a broad range of other linguistic and psycholinguistic phenomena (Lund, Burgess, and Atchley 1995; Landauer and Dumais 1997; Baroni and Lenci 2010; Erk, Pad´o, and Pad´o 2010).”
De-Conflated Semantic Representations (2016, methodology): “However, our approach is equally applicable to any count-based representation technique (Baroni and Lenci, 2010; Turney and Pantel, 2010) or any other embedding approach (Pennington et al., 2014; LeCun et al., 2015).”
Unsupervised Cross-Domain Word Representation Learning (2015, background): “Unlike in domain adaptation, where there is a clear distinction between the source (i.e. the domain on which we train) vs. the target (i.e. the domain on which we test) domains, for representation learning purposes we do not make a distinction between the two domains.”