Know thy Corpus! Robust Methods for Digital Curation of Web corpora

By Serge Sharoff. 2020

10 citations recorded by the source corpus.

This paper proposes a novel framework for digital curation of Web corpora in order to provide robust estimation of their parameters, such as their composition and the lexicon. In recent years language models pre-trained on large corpora emerged as clear winners in numerous NLP tasks, but no proper analysis of the corpora which led to their success has been conducted. The paper presents a procedure for robust frequency estimation, which helps in establishing the core lexicon for a given corpus, as well as a procedure for estimating the corpus composition via unsupervised topic models and via supervised genre classification of Web pages. The results of the digital curation study applied to several Web-derived corpora demonstrate their considerable differences. First, this concerns different frequency bursts which impact the core lexicon obtained from each corpus. Second, this concerns the kinds of texts they contain. For example, OpenWebText contains considerably more topical news and political argumentation in comparison to ukWac or Wikipedia. The tools and the results of analysis have been released.

Subfields: Artificial Intelligence, Communication, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, methodology): “To allow for further analyses, we parse the URLs given with OWTC documents to extract the domain (often a news website, Figure 5 in Appendix §D; Sharoff, 2020), which we crossreference with news factuality ratings by Baly et al. (2018).”
Impact of Pretraining Term Frequencies on Few-Shot Reasoning (2022, background): “Despite this, researchers have been able to estimate word frequencies, topics, and genres of documents (Sharoff, 2020), as well as proportions of toxic text (Gehman et al.”