Comparing Corpora using Frequency Profiling

By Paul Rayson, R. Garside. Proceedings of the workshop on Comparing corpora -, 2000

669 citations recorded by the source corpus.

This paper describes a method of comparing corpora which uses frequency profiling. The method can be used to discover key words in the corpora which differentiate one corpus from another. Using annotated corpora, it can be applied to discover key grammatical or word-sense categories. This can be used as a quick way in to find the differences between the corpora and is shown to have applications in the study of social differentiation in the use of English vocabulary, profiling of learner English and document analysis in the software engineering process.

Subfields: Artificial Intelligence, Developmental and Educational Psychology, Language and Linguistics

View Paper: https://www.semanticscholar.org/paper/a290f9b683b52988ecdd20ddbf37acbc6c3b4040

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Developmental and Educational Psychology

In Language and Linguistics

Citations in the corpus

Introducing, evaluating ukWaC, a very large web-derived corpus of English (2008, background): “7 This makes it possible to identify the words that are most characteristic of one corpus when compared to the other (Rayson and Garside, 2000).”
Dynamic Wordclouds and Vennclouds for Exploratory Data Analysis (2014, background): “The dynamic wordcloud was implicitly called for in (Rayson and Garside, 2000) since human expertise (specifically knowledge of broader contexts and common sense) is needed to separate meaningful and non-meaningful differences in wordclouds.”
BootCaT: Bootstrapping Corpora and Terms from the Web (2004, neutral): “See, for example, Rayson and Garside (2000).”
An Academic Formulas List: New Methods in Phraseology Research (2010, background): “The LL ratio is useful for comparing the relative frequency of words or phrases across registers and determining whether the frequency of an item is statistically higher in one corpus or subcorpus than another (Oakes 1998; Jurafsky and Martin 2000; Rayson and Garside 2000).”
Know thy Corpus! Robust Methods for Digital Curation of Web corpora (2020, methodology): “The effect of Winsorisation can be measured by using the log-likelihood (LL) score (Rayson and Garside, 2000), i.e., by comparing the original frequency counts against the robust frequency counts from the same corpus as follows: LL = R ln R E + C ln C E ;where E = C +R 2 Words affected by the…”
Measuring the Diversity of Automatic Image Descriptions (2018, methodology): “Using the coverage metric to evaluate the nine systems, we find that the GAN-based systems of Shetty et al. (2017) and Dai et al.”
A Corpus-Based Sociolinguistic Study of Amplifiers in British English (2007, methodology): “As noted in Section 1, the question of which gender uses amplifiers more frequently has been a long standing issue since Stoffel (1901), Jespersen (1922) and especially Lakoff ’s work in the 1970s, and competing and conflicting answers have been proposed to this question.”
Why we twitter: understanding microblogging usage and communities (2007, methodology): “Using a technique described by Rayson and Garside [30], we create a contingency table of term frequencies for a given day and the rest of the week.”