BARTScore: Evaluating Generated Text as Text Generation
By Weizhe Yuan, Graham Neubig, Pengfei Liu. 2021
1,107 citations recorded by the source corpus.
A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with a number of variants that can be flexibly applied in an unsupervised fashion to evaluation of text from different perspectives (e.g. informativeness, fluency, or factuality). BARTScore is conceptually simple and empirically effective. It can outperform existing top-scoring metrics in 16 of 22 test settings, covering evaluation of 16 datasets (e.g., machine translation, text summarization) and 7 different perspectives (e.g., informativeness, factuality). Code to calculate BARTScore is available at https://github.com/neulab/BARTScore, and we have released an interactive leaderboard for meta-evaluation at http://explainaboard.nlpedia.ai/leaderboard/task-meval/ on the ExplainaBoard platform, which allows us to interactively understand the strengths, weaknesses, and complementarity of each metric.
Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Graham Neubig
- Percy Liang
- Yejin Choi
- Caiming Xiong
- Noah A. Smith
In Information Systems
- Percy Liang
- Graham Neubig
- Luke S. Zettlemoyer
- Dan Roth
- Dragomir R. Radev
- Wenhu Chen
- Jiawei Han
- Tong Zhang
In Evaluation & Benchmarks
- Luke S. Zettlemoyer
- Caiming Xiong
- Mohit Bansal
- Pengfei Liu
- Jianfeng Gao
- Graham Neubig
- Dan Roth
- Percy Liang
In Natural Language Generation
- William Yang Wang
- Lei Li
- Wenda Xu
- Xiang Ren
- Bill Yuchen Lin
- Ajit Rajasekharan
- Dan Jurafsky
- Sijia Wang
Citations in the corpus
- GPTScore: Evaluate as You Desire (2023, methodology): “Yuan et al. (2021) demonstrates the feasibility of using the probability of text generation as a text quality score, and fine-tuning is required to achieve better performance.”
- Natural Language to Code Translation with Execution (2022, background): “…text classification (Shin et al., 2020; Gao et al., 2021; Min et al., 2021, inter alia ), question answering (Khashabi et al., 2020), machine translation (Radford et al., 2019), and evaluation of generated text (Yuan et al., 2021), where no more than a few examples are used to construct the prompts.”
- Benchmarking Large Language Models for News Summarization (2023, methodology): “The list of metrics we evaluate are: Rouge-L (Lin, 2004), METEOR (Baner-jee and Lavie, 2005), BertScore (Zhang* et al., 2020), BLEURT (Sellam et al., 2020), and BARTScore (Yuan et al., 2021). on CNN/DM and XSUM so we discuss them separately in the following paragraphs.”
- Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization (2021, background): “Yuan et al. (2021) more recently propose BARTScore and meta evaluate it on multiple evaluation perspectives.”
- Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “BARTScore (Yuan et al., 2021) scores generated text by its log probability given another text.”
- From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024, positive): “With the rise of deep learning models (Devlin et al., 2019; Reimers and Gurevych, 2019), many embedding-based assessment methods (e.g., BERTScore (Zhang et al., 2020) and BARTScore (Yuan et al., 2021)) have also emerged.”
- MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text (2022, methodology): “For evaluation, WebQA uses BARTScore (Yuan et al., 2021) to measure the fluency between the generation and the reference,”
- Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, methodology): “On the other hand, several studies worked on building a unified evaluator, i.e., a single model that can produce multiple metrics (e.g., precision and recall) for the generated text (Yuan et al., 2021).”
- PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, methodology): “The answers are long-form and measured by two scores: the fluency score measured by BARTScore [80], and the accuracy score that measures if human-annotated keywords are included in the answer.”
- SMART: Sentences as Basic Units for Text Evaluation (2022, methodology): “More recently, pretrained language models are leveraged for automatically evaluating system-generated texts (Zhang* et al., 2020; Sellam et al., 2020; Yuan et al., 2021), which have shown improvements on correlation with human judgments.”
- SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation (2022, methodology): “, 2006); 2) four best performed learned metrics without human ratings: PRISM (Thompson and Post, 2020), BARTScore (Yuan et al., 2021), BERTScore (Zhang et al.”
- RAGAs: Automated Evaluation of Retrieval Augmented Generation (2023, methodology): “BARTScore (Yuan et al., 2021) similarly uses reference answers to compute aspects such as precision (estimated as the probability of generating the generated answer given the reference) and recall (estimated as the probability of generating the reference given the generated answer).”
- Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis (2022, positive): “They are (1) Ngram-and distance-based metrics (BLEU (Papineni et al., 2002), ChrF (Popovi´c, 2015) and TER (Snover et al., 2006)); (2) learned metrics requiring human rating data (COMET (Rei et al., 2020), BLEURT (Sellam et al., 2020)); (3) learned metrics without human rating data (PRISM (Thompson ”
- Black-box Prompt Learning for Pre-trained Language Models (2022, background): “…prompts, the prompt-based learning can be categorized into two kinds: discrete prompt (Wallace et al., 2019; Shin et al., 2020; Jiang et al., 2020; Yuan et al., 2021; Haviv et al., 2021; Gao et al., 2021; Ben-David et al., 2021) and continuous prompt (Zhong et al., 2021; Qin and Eisner, 2021;…”
- Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, methodology): “The prompts can be combined using a uniform average [75, 166, 210] or a weighted average [75, 143, 166, 167].”