Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset

By Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, Daniel E. Ho. 2022

154 citations recorded by the source corpus.

One concern with the rise of large language models lies with their potential for significant harm, particularly from pretraining on biased, obscene, copyrighted, and private information. Emerging ethical approaches have attempted to filter pretraining material, but such approaches have been ad hoc and failed to take context into account. We offer an approach to filtering grounded in law, which has directly addressed the tradeoffs in filtering material. First, we gather and make available the Pile of Law, a 256GB (and growing) dataset of open-source English-language legal and administrative data, covering court opinions, contracts, administrative rules, and legislative records. Pretraining on the Pile of Law may help with legal tasks that have the promise to improve access to justice. Second, we distill the legal norms that governments have developed to constrain the inclusion of toxic or private content into actionable lessons for researchers and discuss how our dataset reflects these norms. Third, we show how the Pile of Law offers researchers the opportunity to learn such filtering rules directly from the data, providing an exciting new research direction in model-based processing.

Subfields: Political Science and International Relations, Large Language Models, AI Ethics, Fairness & Society, Social Sciences & Humanities

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Political Science and International Relations

In Large Language Models

In AI Ethics, Fairness & Society

In Social Sciences & Humanities

Citations in the corpus

Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, neutral): “, 2021) Privacy Filtering private/duplicate data (Henderson et al., 2022; Kandpal et al., 2022; Lee et al., 2022b)”
Foundation Models and Fair Use (2023, background): “For example, Henderson et al. (2022) suggest that a contextual approach is needed to filter input content for privacy and toxicity.”
The future landscape of large language models in medicine (2023, background): “Popular applications of LLMs in the ﬁ elds of data science and bioinformatics are code debugging and simpli ﬁ cation, translation to different programming languages, and derivation of code from natural language input (Supplementary Data, example 7).”
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (2023, methodology): “The text generally falls into eight different domains: • pd by Legal: We curate legal text from the Pile of Law (Henderson et al., 2022), an amalgation of 31 different sources of text related to civil court cases, patents, and other legal and governmental works, either licensed as public domain or C”
Scaling Data-Constrained Language Models (2023, background): “[38] Peter Henderson, Mark Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel Ho.”
Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (2024, unknown): “He then left the building.”
The MiniPile Challenge for Data-Efficient Language Models (2023, background): “Potential issues with such include token repetitions [33, 21, 1], misogyny, pornography, and malignant stereotypes [9, 7], benchmark data contamination [11, 14], spurious correlations [27, 37, 38], diluted robustness due to data mixing [45] and potentially sensitive information [20].”