AlignScore: Evaluating Factual Consistency with A Unified Alignment Function

By Yuheng Zha, Yichi Yang, Ruichen Li, Zhiting Hu. Annual Meeting of the Association for Computational Linguistics, 2023

378 citations recorded by the source corpus.

Many text generation applications require the generated text to be factually consistent with input information. Automatic evaluation of factual consistency is challenging. Previous work has developed various metrics that often depend on specific functions, such as natural language inference (NLI) or question answering (QA), trained on limited data. Those metrics thus can hardly assess diverse factual inconsistencies (e.g., contradictions, hallucinations) that occur in varying inputs/outputs (e.g., sentences, documents) from different tasks. In this paper, we propose AlignScore, a new holistic metric that applies to a variety of factual inconsistency scenarios as above. AlignScore is based on a general function of information alignment between two arbitrary text pieces. Crucially, we develop a unified training framework of the alignment function by integrating a large diversity of data sources, resulting in 4.7M training examples from 7 well-established tasks (NLI, QA, paraphrasing, fact verification, information retrieval, semantic similarity, and summarization). We conduct extensive experiments on large-scale benchmarks including 22 evaluation datasets, where 19 of the datasets were never seen in the alignment training. AlignScore achieves substantial improvement over a wide range of previous metrics. Moreover, AlignScore (355M parameters) matches or even outperforms metrics based on ChatGPT and GPT-4 that are orders of magnitude larger.

Subfields: Artificial Intelligence, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/27cb586fcea5ec076b984750e9c77f0d7fc976e5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qun Liu
2. Xin Jiang
3. Lifeng Shang
4. Liangyou Li
5. Xinwei Wu
6. Tianhao Shen
7. Weilong Dong
8. Chuang Liu

In Molecular Biology

1. Qun Liu
2. Yufei Huang

Citations in the corpus

• Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, neutral): “AlignScore (Zha et al., 2023) establishes a unified function to evaluate the factual consistency between two texts.”
• Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, background): “Thereafter, some novel metrics have been proposed to detect hallucination errors, such as AlignScore in [81].”
• Aligning Large Language Models with Human: A Survey (2023, background): “Min et al. (2023); Zha et al. (2023) propose to evaluate factual correctness using both closedsourced and open-sourced LLMs. Similar to human evaluation, there are also research efforts in explicitly prompting LLMs to conduct pairwise comparisons.”
• Large Language Model Alignment: A Survey (2023, background): “The ALIGNSCORE metric, proposed by (Zha et al., 2023), is designed to cover a wide range of factual consistency evaluation scenarios, such as contradiction and hallucination across various lengths and tasks.”
• Automated Evidence Extraction and Scoring for Corporate Climate Policy Engagement: A Multilingual RAG Approach (2025, unknown): “Faithfulness 2 (Zha et al., 2023) and conciseness 3 scores, s f and s c respectively, compare the alignment of the retrieved snip-pet e with the gold snippet G .”