From Frequency to Meaning: Vector Space Models of Semantics

By Peter D. Turney, Patrick Pantel. 2010

3,105 citations recorded by the source corpus.

Computers understand very little of the meaning of human language. This profoundly limits our ability to give instructions to computers, the ability of computers to explain their actions to us, and the ability of computers to analyse and process text. Vector space models (VSMs) of semantics are beginning to address these limits. This paper surveys the use of VSMs for semantic processing of text. We organize the literature on VSMs according to the structure of the matrix in a VSM. There are currently three broad classes of VSMs, based on term-document, word-context, and pair-pattern matrices, yielding three classes of applications. We survey a broad range of applications in these three categories and we take a detailed look at a specific open source project in each category. Our goal in this survey is to show the breadth of applications of VSMs for semantics, to provide a new perspective on VSMs for those who are already familiar with the area, and to provide pointers into the literature for those who are less familiar with the field.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Distributed Representations of Sentences and Documents (2014, neutral): “…NLP applications such as word representation, named entity recognition, word sense disam-biguation, parsing, tagging and machine translation (Col-lobert & Weston, 2008; Turney & Pantel, 2010; Turian et al., 2010; Collobert et al., 2011; Socher et al., 2011b; Huang et al., 2012; Zou et al., 2013).”
Distributed Representations of Words and Phrases and their Compositionality (2013, methodology): “to statistical language modeling with considerable success [1]. The follow up work includes applications to automatic speech recognition and machine translation [14, 7], and a wide range of NLP tasks [2, 20, 15, 3, 18, 19, 9]. Recently, Mikolov et al. [8] introduced the Skip-gram model, an efﬁcient ”
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank (2013, background): “Often, co-occurrence statistics of a word and its context are used to describe each word (Turney and Pantel, 2010; Baroni and Lenci, 2010), such as tf-idf.”
Experience Grounds Language (2020, background): “The intuition of such embedding representations, that context lends meaning, has long been acknowledged (Firth, 1957; Turney and Pantel, 2010).”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, background): “Often, co-occurrence statistics of a word and its context are used to describe each word (Turney and Pantel, 2010; Baroni and Lenci, 2010), such as tf-idf.”
Zero-Shot Learning Through Cross-Modal Transfer (2013, background): “In these approaches, words are represented as vectors of distributional characteristics – most often their co-occurrences with words in context [26, 9, 1, 32].”
Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change (2016, methodology): “The semantic similarity/distance be-tween two words is approximated by the cosine similarity/distance between their vectors (Turney and Pantel, 2010).”
Semantics derived automatically from language corpora contain human-like biases (2016, background): “A word embedding is a representation of words as points in a vector space (12).”
Grounding Action Descriptions in Videos (2013, background): “Such distributional models are attractive because they are conceptually simple, easy to implement and relevant for various NLP tasks (Turney and Pantel, 2010).”
Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “Word embeddings have been responsible for state-of-the-art results in a wide range of NLP tasks [9, 10, 11, 12].”
Leverage Financial News to Predict Stock Price Movements Using Word Embeddings and Deep Neural Networks (2015, background): “(2) Polarity score (PS): We further compute socalled polarity scores (Turney and Littman, 2003; Turney and Pantel, 2010) to measure how each keyword is related to stock movements and how each keyword applies to a target stock in each sentence.”
Improving Distributional Similarity with Lessons Learned from Word Embeddings (2015, background): “A well-known shortcoming of PMI, which persists in PPMI, is its bias towards infrequent events (Turney and Pantel, 2010).”
Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors (2014, methodology): “…track of the contexts (e.g., co-occurring words) in which target terms appear in a large corpus as proxies for meaning representations, and apply geometric techniques to these vectors to measure the similarity in meaning of the corresponding words (Clark, 2013; Erk, 2012; Turney and Pantel, 2010).”
A Roadmap Towards Machine Intelligence (2015, background): “…notice that once the machine can read text, it can also exploit distributional learning from large amounts of text (Erk, 2012; Mikolov et al., 2013; Turney and Pantel, 2010) to induce word and phrase representations addressing some of the challenging natural language phenomena we just mentioned,…”
Multimodal Distributional Semantics (2013, methodology): “They approximate the meaning of words with vectors that record their distributional history in a corpus (Turney & Pantel, 2010).”
Improving Vector Space Word Representations Using Multilingual Correlation (2014, background): “Using cooccurrence statistics from a large corpus of text (Deerwester et al., 1990; Turney and Pantel, 2010), 1 it is possible to construct high-quality semantic vectors — as judged by both correlations with human judgements of semantic relatedness (Turney, 2006; Agirre et al., 2009) and as…”