Mining the Web for Synonyms: PMI-IR versus LSA on TOEFL

By Peter D. Turney. European Conference on Machine Learning, 2001

1,513 citations recorded by the source corpus.

This paper presents a simple unsupervised learning algorithm for recognizing synonyms, based on statistical data acquired by querying a Web search engine. The algorithm, called PMI-IR, uses Pointwise Mutual Information (PMI) and Information Retrieval (IR) to measure the similarity of pairs of words. PMI-IR is empirically evaluated using 80 synonym test questions from the Test of English as a Foreign Language (TOEFL) and 50 synonym test questions from a collection of tests for students of English as a Second Language (ESL). On both tests, the algorithm obtains a score of 74%. PMI-IR is contrasted with Latent Semantic Analysis (LSA), which achieves a score of 64% on the same 80 TOEFL questions. The paper discusses potential applications of the new unsupervised learning algorithm and some implications of the results for LSA and LSI (Latent Semantic Indexing).

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/e517e1645708e7b050787bb4734002ea194a1958

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

From Frequency to Meaning: Vector Space Models of Semantics (2010, methodology): “A variation of PMI is Positive PMI (PPMI), in which all PMI values that are less than zero are replaced with zero (Niwa & Nitta, 1994).”
A survey of named entity recognition and classification (2007, methodology): “PMI-IR, developed by P. Turney (2001), measures the dependence between two expressions using web queries.”
Corpus-based and Knowledge-based Measures of Text Semantic Similarity (2006, methodology): “In a set of experiments based on TOEFL synonymy tests (Turney 2001), the PMI-IR measure using the NEAR operator accurately identified the correct answer (out of four synonym choices) in 72.”
Verb similarity on the taxonomy of WordNet (2006, background): “Knowledge-poor methods mainly depend on information or probability information derived from a corpus or the Internet (Turney, 2001) rather than a knowledge base.”
Neural Network Methods for Natural Language Processing (2017, background): “3See the survey of Turney and Pantel [2010] and Baroni and Lenci [2010] for an overview.”
VerbOcean: Mining the Web for Fine-Grained Semantic Verb Relations (2004, background): “Turney (2001) studied word relatedness and synonym extraction, while Lin et al. (2003) present an algorithm that queries the Web using lexical patterns for distinguishing noun synonymy and antonymy.”
WebTables: exploring the power of tables on the web (2008, methodology): “Turney used language cooccurrence statistics from the Web to answer standardized-test-style synonym questions, but relies on wordproximity statistics that seem inapplicable to structured data [25].”
Roget's thesaurus and semantic similarity (2012, methodology): “…1998; Hirst and St-Onge, 1998; Leacock and Chodorow, 1998) and Roget’s Thesaurus (McHale, 1998), or statistical methods (Landauer and Dumais, 1997; Turney, 2001; Bigham et al., 2003) We set out to test the intuition that Roget’s Thesaurus, sometimes treated as a book of synonyms, allows us to…”
Measuring semantic similarity between words using web search engines (2007, methodology): “Turney [Turney 2001] defined a point-wise mutual information (PMI-IR) measure using the number of hits returned by a web search engine to recognize synonyms.”
Sentiment Analysis of Short Informal Texts (2014, methodology): “We have chosen PMI because it is simple and robust and has been successfully applied in a number of NLP tasks (Turney, 2001; Turney & Littman, 2003).”
Deep learning meets ontologies: experiments to anchor the cardiovascular disease ontology in the biomedical literature (2018, background): “Two well-known datasets for evaluating synonymy are the 80 TOEFL (Test of English as a Foreign Language) synonym questions from [21] and the 50 ESL (English as a Second Language) synonym questions from [44].”
An Improved Model of Semantic Similarity Based on Lexical Co-Occurrence (2005, methodology): “The second vocabulary test consists of 50 items drawn from the English as a Second Language (ESL) test (Turney, 2001).”
Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning (2011, methodology): “Inspired by the approach of Turney (2001), we replaced the frequency counts computed over the text collection in the first baseline with the number of hits retrieved by a web search engine.”
Neural Word Embedding as Implicit Matrix Factorization (2014, methodology): “PMI can be estimated empirically by considering the actual number of observations in a corpus: The use of PMI as a measure of association in NLP was introduced by Church and Hanks [8] and widely adopted for word similarity tasks [11, 27, 29].”