Word Association Norms, Mutual Information, and Lexicography

By Kenneth Ward Church, Patrick Hanks. 1989

The term word association is used in a very particular sense in the psycholinguistic literature. (Generally speaking, subjects respond quicker than normal to the word nurse if it follows a highly associated word such as doctor. ) We will extend the term to provide the basis for a statistical description of a variety of interesting linguistic phenomena, ranging from semantic relations of the doctor/nurse type (content word/content word) to lexico-syntactic co-occurrence constraints between verbs and prepositions (content word/function word). This paper will propose an objective measure based on the information theoretic notion of mutual information, for estimating word association norms from computer readable corpora. (The standard method of obtaining word association norms, testing a few thousand subjects on a few hundred words, is both costly and unreliable.) The proposed measure, the association ratio, estimates word association norms directly from computer readable corpora, making it possible to estimate norms for tens of thousands of words.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (2018, methodology): “Given a question q and an answer option a i , it uses PMI or point-wise mutual information (Church and Hanks, 1989) to measure the strength of the associations between parts of q and parts of a i .”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, methodology): “We extract the frequencies of a set of 22 regular expressions related to identity mentions,21 and compute the pointwise mutual information (PMI; Church and Hanks, 1990) between the likelihood of an identity mention occurring versus being filtered out by the blocklist.”
Social Bias in Elicited Natural Language Inferences (2017, background): “Though we recognize these sources of bias may not be as tidy and independent as their names suggest, it is a useful conceptual shorthand: In this paper, we are primarily interested in detecting elicitation bias.”
From Frequency to Meaning: Vector Space Models of Semantics (2010, background): “A variation of PMI is Positive PMI (PPMI), in which all PMI values that are less than zero are replaced with zero (Niwa & Nitta, 1994).”
Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions (2016, methodology): “Given a question q and an answer option a i , it uses pointwise mutual information (Church and Hanks 1989) to measure the strength of the associations between parts of q and parts of a i .”
Improving Distributional Similarity with Lessons Learned from Word Embeddings (2015, neutral): “Nevertheless, the PPMI measure is widely regarded as state-of-the-art for these kinds of distributional-similarity models.”
Improving Vector Space Word Representations Using Multilingual Correlation (2014, methodology): “We then replace every entry in the sparse frequency matrix F by its pointwise mutual information (PMI) (Church and Hanks, 1990; Turney, 2001) resulting in X .”
Linguistic Regularities in Sparse and Explicit Word Representations (2014, methodology): “The use of PMI in distributional similarity models was introduced by Church and Hanks (1990) and widely adopted (Dagan et al.”
Distributional Memory: A General Framework for Corpus-Based Semantics (2010, positive): “We compute the pointwise mutual information (PMI) statistic (Church and Hanks 1990) on these lengths to ﬁnd the links that are most typical of the positive subspace corresponding to each qualia role.”
Deep Neural Networks for Learning Graph Representations (2016, background): “Church and Hanks’s pointwise mutual information (PMI) matrix (Church and Hanks 1990) was proposed to address this problem and has since been proven to provide better word representations (Bullinaria and Levy 2007):”
Bootstrapping Path-Based Pronoun Resolution (2006, methodology): “Thus we attempted to duplicate Bergsma’s corpus-based extraction of gender and number, where the information can be stored in advance in a table, but using a much larger data set.”