Cross-Policy Compliance Detection via Question Answering

By Marzieh Saeidi, Majid Yazdani, Andreas Vlachos. 2021

11 citations recorded by the source corpus.

Policy compliance detection is the task of ensuring that a scenario conforms to a policy (e.g. a claim is valid according to government rules or a post in an online platform conforms to community guidelines). This task has been previously instantiated as a form of textual entailment, which results in poor accuracy due to the complexity of the policies. In this paper we propose to address policy compliance detection via decomposing it into question answering, where questions check whether the conditions stated in the policy apply to the scenario, and an expression tree combines the answers to obtain the label. Despite the initial upfront annotation cost, we demonstrate that this approach results in better accuracy, especially in the cross-policy setup where the policies during testing are unseen in training. In addition, it allows us to use existing question answering models pre-trained on existing large datasets. Finally, it explicitly identifies the information missing from a scenario in case policy compliance cannot be determined. We conduct our experiments using a recent dataset consisting of government policies, which we augment with expert annotations and find that the cost of annotating question answering decomposition is largely offset by improved inter-annotator agreement and speed.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “Finally, a key practical advantage of collecting data via detailed rules is that conflicts and weighting between rules can be changed after the fact: Saeidi et al. (2021) express policies as expression trees with rules as the leaves, with the expression either written by experts or inferred from…”
Policy Compliance Detection via Expression Tree Inference (2022, methodology): “Previous work (Holzenberger et al., 2020) has formulated the PCD problem in the style of Recognizing Textual Entailment (Dagan et al., 2005), i.e. the answer is the inferred entailment relation between the policy (text) and the scenario (hypothesis).”