Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning

By Vivek Gupta, Shuo Zhang, Alakananda Vempala, Yujie He, Temma Choji, Vivek Srikumar. 2022

18 citations recorded by the source corpus.

When pre-trained contextualized embedding-based models developed for unstructured data are adapted for structured tabular data, they perform admirably. However, recent probing studies show that these models use spurious correlations, and often predict inference labels by focusing on false evidence or ignoring it altogether. To study this issue, we introduce the task of Trustworthy Tabular Reasoning, where a model needs to extract evidence to be used for reasoning, in addition to predicting the label. As a case study, we propose a two-stage sequential prediction approach, which includes an evidence extraction and an inference stage. First, we crowdsource evidence row labels and develop several unsupervised and supervised evidence extraction strategies for InfoTabS, a tabular NLI benchmark. Our evidence extraction strategy outperforms earlier baselines. On the downstream tabular inference task, using only the automatically extracted evidence as the premise, our approach outperforms prior benchmarks.

Subfields: Artificial Intelligence, Prompting & Reasoning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Murray Shanahan
2. Antonia Creswell
3. Kevin Chen-Chuan Chang
4. Jie Huang

In Prompting & Reasoning

1. Murray Shanahan
2. Antonia Creswell
3. Jie Huang
4. Kevin Chen-Chuan Chang

In Safety, Robustness & Interpretability

1. Murray Shanahan
2. Antonia Creswell

Citations in the corpus

• Faithful Reasoning Using Large Language Models (2022, background): “However, while Gupta et al. (2022) and Nakano et al. (2021) show the evidence used, they do not show how that information was combined to answer the question.”