Using Statistics in Lexical Analysis

By Kenneth Ward Church, William A. Gale, Patrick Hanks, Donald Hindle. Lexical Acquisition: Exploiting On-Line Resources to Build a Lexicon, 2003

446 citations recorded by the source corpus.

The computational tools available for studying machine-readable corpora are at present still rather primitive. In the more advanced lexicographic organizations, there are concordancing programs (see figure below), which are basically KWIC (key word in context (Aho et al., 1988, p. 122), (Salton, 1989, p. 384)) indexes with additional features such as the ability to extend the context, sort leftwards as well as rightwards, and so on. There is very little interactive software. The lack of interactive software is perhaps part of the reason why dictionaries produced in the United States pay little attention to machine-readable corpora, and are based on collections of selected citations, augmented by introspection, rather than analysis of whole texts. The situation is somewhat different in Britain. British lexicographers, especially those working on dictionaries for foreign learners, are beginning to depend heavily on machine-readable corpora. They use these corpora and the basic concordancing tool mentioned above to fill in detailed syntactic descriptions (prompting a move, that will probably dominate lexicography in the 1990s, towards more thorough descriptions of lexical syntax). In the Cobuild project of the 1980s, for example, the typical procedure was that a lexicographer was given the concordances for a word or group of words, marked up the printout with colored pens in order to identify the salient senses, and then wrote syntactic descriptions and definitions.

Subfields: Language and Linguistics

View Paper: https://www.semanticscholar.org/paper/ccd5898c79f90c25e24222410d5613edfd007985

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Language and Linguistics

Citations in the corpus

Collostructions: Investigating the interaction of words and constructions (2003, background): “degrees of collocational strength) (Church et al. 1990, 1991, 1994). However, while the basic argument is by now generally accepted, it is far from clear which method is optimally suited for linguistic research, and Church et al.’s work has triggered a number of studies proposing a variety of measur”
Topical N-Grams: Phrase and Topic Discovery, with an Application to Information Retrieval (2007, methodology): “More recently, an information-theoretically motivated method for collocation discovery is utilizing mutual information [ 6 , 11].”
A Language Model Approach to Keyphrase Extraction (2003, methodology): “Word collocation Various collocation metrics have been proposed, including mean and variance (Smadja, 1994), the t-test (Church et al., 1991), the chi-square test, pointwise mutual information (MI) (Church and Hanks, 1990), and binomial loglikelihood ratio test (BLRT) (Dunning, 1993).”
The Statistics of Word Cooccur-rences: Word Pairs and Collocations (2004, methodology): “Among the best-known measures are MI (Mututal Information, an information-theoretic notion suggested by Church and Hanks (1990)), the t-score measure t (Church et al. 1991), the log-likelihood ratio G2 (Dunning 1993), and to a lesser extent also the chi-squared statistic X2, which is the standard me”
The meanings of ‘risk’: a view from corpus linguistics (2007, background): “In terms of statistical measures two of the more common calculations are the Mutual Information score (MI score) (see Church and Hanks, 1989), t-score (see Church et al., 1991) and log-likelihood (Dunning, 1993).”
Mining the Web for Synonyms: PMI-IR versus LSA on TOEFL (2001, methodology): “PMI-IR uses Pointwise Mutual Information (PMI) [1, 2], as follows:”
Extending collostructional analysis: A corpus-based perspective on `alternations' (2004, background): “The logic behind this extension is basically that of Church et al.’s (1991) distinctive collocate analysis, which uses a”