TRUE: Re-evaluating Factual Consistency Evaluation

By Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, Yossi Matias. 2022

372 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Summarization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Summarization

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, methodology): “To assess the example-level accuracy of metrics in diverse tasks, TRUE [72] reports their Area Under the ROC Curve (ROC AUC) in regard to hallucinated example detection.”
Making Retrieval-Augmented Language Models Robust to Irrelevant Context (2023, methodology): “Building on the strong performance of recent NLI models (e.g., in detecting model hallucinations (Honovich et al., 2022) and attributed question answering (Bohnet et al., 2023)), we use such models to identify irrelevant contexts.”
SMART: Sentences as Basic Units for Text Evaluation (2022, methodology): “For T5-ANLI, we used the same implementation as in Honovich et al. (2022), where T5-11B is ﬁne-tuned with 25K training steps on ANLI (Nie et al., 2020), treating both contradiction and neutral pairs as not entailed.”
ExpertQA: Expert-Curated Questions and Attributed Answers (2023, methodology): “For automatic attribution with AutoAIS, we use the t5_xxl_true_nli_mixture 6 with 11B parameters by Honovich et al. (2022).”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “, 2022b), including those specific to factual consistency evaluation (Kryscinski et al., 2020; Durmus et al., 2020; Wang et al., 2020; Maynez et al., 2020; Laban et al., 20d; Fabbri et al., 2022b; Honovich et al., 2022; Tam et al., 2022).”
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023, methodology): “, 2019) for measuring n-gram lexical entailment in table-totext generation and TRUE (Honovich et al., 2022) computes the example-level Area Under the ROC Curve.”
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, methodology): “The results on the TRUE benchmark are shown in Table 3, where ALIGNSCORE-large gets the highest average AUC-ROC score.”
Enabling Large Language Models to Generate Text with Citations (2023, methodology): “We use the version of TRUE model from https://huggingface.co/google/ t5_xxl_true_nli_mixture , which is trained on SNLI (Bowman et al., 2015), MNLI (Williams et al., 2018), Fever (Thorne et al., 2018), Sci-tail (Khot et al., 2018), PAWS (Zhang et al., 2019), and VitaminC (Schuster et al., 2021).”
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model (2022, methodology): “This style of evaluation has been applied to other tasks involving factual consistency, such as summarization and fact veriﬁcation (Honovich et al., 2022).”
A Survey on Evaluation of Large Language Models (2023, neutral): “, 2023a) X (Honovich et al., 2022) X X X X (Pezeshkpour, 2023) X (Gekhman et al.”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, methodology): “For this, we use a version of the E2E NLI model from Honovich et al. (2022).4 We refer to this model as auto-AIS.”
BITEM at the NTCIR-19 R2C2 Task: Predicting Confidence from Agentic RAG Pipeline Signals (2026, unknown): “Honovich et al. [3] evaluate such checking across nine grounding datasets, where the best entailment model reaches 81.5 ROC AUC against 63.8 for token-level F1.”