Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction
By Adrien Barbaresi. 2021
193 citations recorded by the source corpus.
An essential operation in web corpus construction consists in retaining the desired content while discarding the rest. Another challenge finding one’s way through websites. This article introduces a text discovery and extraction tool published under open-source license. Its installation and use is straightforward, notably from Python and on the command-line. The software allows for main text, comments and metadata extraction, while also providing building blocks for web crawling tasks. A comparative evaluation on real-world data also shows its interest as well as the performance of other available solutions. The contributions of this paper are threefold: it references the software, features a benchmark, and provides a meaningful baseline for similar tasks. The tool performs significantly better than other open-source solutions in this evaluation and in external benchmarks.
Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Information Extraction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Colin Raffel
- Mohit Bansal
- Thomas Wolf
- Jianfeng Gao
- Luke S. Zettlemoyer
- Yejin Choi
- Hannaneh Hajishirzi
- Noah A. Smith
In Information Systems
- Ruxandra-Aimée Cojocaru
- Julien Launay
- Quentin Malartic
- Ebtesam Almazrouei
- Alessandro Cappelli
- Zhangir Azerbayev
- Anton Lozhkov
- Guilherme Penedo
In Natural Language Processing
- Thomas Wolf
- Colin Raffel
- Ruxandra-Aimée Cojocaru
- Julien Launay
- Quentin Malartic
- Ebtesam Almazrouei
- Alessandro Cappelli
- Zhangir Azerbayev
In Information Extraction
- Ruxandra-Aimée Cojocaru
- Julien Launay
- Quentin Malartic
- Ebtesam Almazrouei
- Alessandro Cappelli
- Zhangir Azerbayev
- Anton Lozhkov
- Guilherme Penedo
Citations in the corpus
- The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “We first filter URLs to remove adult content using a blocklist and a score sys-tem (Appendix G.1), we then use trafilatura (Barbaresi, 2021) to extract content from pages, and perform language identification with the fastText classifier from CCNet (Wenzek et al., 2020).”
- OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text (2023, methodology): “Likewise, Trafilatura (Barbaresi, 2021), which was used in RefinedWeb (Penedo et al., 2023), had poor efficiency.”
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (2024, methodology): “We therefore experimented with extracting the text content from the WARC files using the open source trafilatura library [49], which from visual inspection of the results provided good quality extraction when compared to other available libraries (less boilerplate and menu text).”
- WebKnoGraph: GNN-Powered Internal Linking (2026, unknown): “During data preparation, boilerplate elements such as navigation menus, footers, and advertisements are removed with trafilatura [7].”