A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering

By Nils Holzenberger, Andrew Blair-Stanek, Benjamin Van Durme. 2020

88 citations recorded by the source corpus.

Legislation can be viewed as a body of prescriptive rules expressed in natural language. The application of legislation to facts of a case we refer to as statutory reasoning, where those facts are also expressed in natural language. Computational statutory reasoning is distinct from most existing work in machine reading, in that much of the information needed for deciding a case is declared exactly once (a law), while the information needed in much of machine reading tends to be learned through distributional language statistics. To investigate the performance of natural language understanding approaches on statutory reasoning, we introduce a dataset, together with a legal-domain text corpus. Straightforward application of machine reading models exhibits low out-of-the-box performance on our questions, whether or not they have been fine-tuned to the legal domain. We contrast this with a hand-constructed Prolog-based system, designed to fully solve the task. These experiments support a discussion of the challenges facing statutory reasoning moving forward, which we argue is an interesting real-world task that can motivate the development of models able to utilize prescriptive rules specified in natural language.

Subfields: Artificial Intelligence, Law, Political Science and International Relations, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Law

In Political Science and International Relations

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Flexible Generation of Natural Language Deductions (2021, background): “Recent datasets have been explicitly designed to test deduction ability (Liu et al., 2020; Yu et al., 2020; Holzenberger et al., 2020) and new types of models take inspiration from formal and informal reasoning (Clark et al.”
“According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data (2023, methodology): “In the legal domain, we use the SARA dataset (Holzenberger et al., 2020) consisting of tax cases to be evaluated using natural language inference.”
Policy Compliance Detection via Expression Tree Inference (2022, background): “Previous work (Holzenberger et al., 2020) has formulated the PCD problem in the style of Recognizing Textual Entailment (Dagan et al., 2005), i.e. the answer is the inferred entailment relation between the policy (text) and the scenario (hypothesis).”
CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review (2021, background): “Holzenberger et al. (2020) introduce a dataset for tax law entailment and question answering and Chalkidis et al. (2019) introduce a large dataset of text classiﬁcation for EU legislation.”
ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts (2021, background): “Holzenberger et al. (2020) introduced a dataset for predicting an entailment relationship between a statement and a statute excerpt.”
Cross-Policy Compliance Detection via Question Answering (2021, methodology): “First we formulate the task as textual entailment following previous work Holzenberger et al. (2020).”