REV: Information-Theoretic Evaluation of Free-Text Rationales
By Hanjie Chen, Hanjie Chen, Faeze Brahman, Xiang Ren, Yangfeng Ji, Yejin Choi, Swabha Swayamdipta. 2022
30 citations recorded by the source corpus.
Generating free-text rationales is a promising step towards explainable NLP, yet evaluating such rationales remains a challenge. Existing metrics have mostly focused on measuring the association between the rationale and a given label. We argue that an ideal metric should focus on the new information uniquely provided in the rationale that is otherwise not provided in the input or the label. We investigate this research problem from an information-theoretic perspective using conditional V-information (Hewitt et al., 2021). More concretely, we propose a metric called REV (Rationale Evaluation with conditional V-information), to quantify the amount of new, label-relevant information in a rationale beyond the information already available in the input or the label. Experiments across four benchmarks with reasoning tasks, including chain-of-thought, demonstrate the effectiveness of REV in evaluating rationale-label pairs, compared to existing metrics. We further demonstrate REV is consistent with human judgments on rationale evaluations and provides more sensitive measurements of new information in free-text rationales. When used alongside traditional performance metrics, REV provides deeper insights into models’ reasoning and prediction processes.
Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Elizabeth Clark
- Sebastian Gehrmann
- Thibault Sellam
- Fan Yang
- Ninghao Liu
- Mengnan Du
- Huiqi Deng
- Shuaiqiang Wang
In Information Systems
- Elizabeth Clark
- Thibault Sellam
- Sebastian Gehrmann
- Mengnan Du
- Ninghao Liu
- Fan Yang
- Dawei Yin
- Hanjie Chen
In Natural Language Processing
- Thibault Sellam
- Elizabeth Clark
- Sebastian Gehrmann
- Huiqi Deng
- Shuaiqiang Wang
- Haiyan Zhao
- Hengyi Cai
- Mengnan Du
In Evaluation & Benchmarks
- Sebastian Gehrmann
- Elizabeth Clark
- Thibault Sellam
- Ninghao Liu
- Fan Yang
- Dawei Yin
- Hanjie Chen
- Mitesh M. Khapra
Citations in the corpus
- Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Similarly, Chen et al. (2021) analyze mispredictions on a set of previously annotated summarization corpora (Kryscinski et al.”
- Explainability for Large Language Models: A Survey (2023, methodology): “Approximation strategies including weighted linear regression, permutation, and other model-specific methods have been adopted (Chen et al., 2023c) to estimate Shapley values.”