CommonCOW: Massively Huge Web Corpora from CommonCrawl Data and a Method to Distribute them Freely under Restrictive EU Copyright Laws
By Roland Schäfer. 2016
22 citations recorded by the source corpus.
In this paper, I describe a method of creating massively huge web corpora from the CommonCrawl data sets and redistributing the resulting annotations in a stand-off format. Current EU (and especially German) copyright legislation categorically forbids the redistribution of downloaded material without express prior permission by the authors. Therefore, such stand-off annotations (or other derivates) are the only format in which European researchers (like myself) are allowed to re-distribute the respective corpora. In order to make the full corpora available to the public despite such restrictions, the stand-off format presented here allows anybody to locally reconstruct the full corpora with the least possible computational effort.
Subfields: Computational Linguistics & Semantics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Linguistics & Semantics
- Gabriel Ilharco
- William Agnew
- Adrien Barbaresi
- Jesse Dodge
- Ana Marasovic
- Dirk Groeneveld
- Matt Gardner
- Margaret Mitchell
Citations in the corpus
- Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, methodology): “Limitations and Recommendations We recognize that we have only examined some of the possible issues with a dataset of this size, and so in addition to making the dataset available to download, we recommend providing a location for others to report issues they find (Habernal et al., 2016; Schäfer, 20”
- Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction (2021, background): “The scraping and conversion parts also work with existing archives, Raw HTML documents can be retrieved from sources such as the CommonCrawl or the Internet Archive 6 .”