BERTScore: Evaluating Text Generation with BERT
By Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi. 2019
9,100 citations recorded by the source corpus.
We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTScore is more robust to challenging examples when compared to existing metrics.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yejin Choi
- Luke S. Zettlemoyer
- Percy Liang
- Graham Neubig
- Noah A. Smith
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Mohit Bansal
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jianfeng Gao
- Cordelia Schmid
In Natural Language Processing
- Mohit Bansal
- Yejin Choi
- Luke S. Zettlemoyer
- Jianfeng Gao
- Graham Neubig
- Percy Liang
- Noah A. Smith
- Hannaneh Hajishirzi
In Evaluation & Benchmarks
- Mohit Bansal
- Yejin Choi
- Elizabeth Clark
- Jianfeng Gao
- Sebastian Gehrmann
- Luke S. Zettlemoyer
- Thibault Sellam
- Hannaneh Hajishirzi
Citations in the corpus
- Automatic Detection of Generated Text is Easiest when Humans are Fooled (2019, methodology): “textual word embeddings from a pre-trained BERT model without fine-tuning to compute a quality score for generated text (Zhang et al., 2020).”
- Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Similarly, Freitag et al. (2020) show that BLEU, METEOR, and BERTSCORE may fail to reward good translations when the reference text contains artifacts such as “translationese”.”
- Learning Compact Metrics for MT (2021, neutral): “5-2X speedup compared to RemBERT-32 by applying length-based batching, a simple optimization which consists in batching examples that have similar a length and cropping the resulting tensor, as done in BERTScore (Zhang* et al., 2020).”
- The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “However, recent work by Fabbri et al. (2020) demonstrates that this may not hold in summarization, where the automated metric BERTScore (Zhang et al., 2020b) does not improve upon the correlation of ROUGE.”
- Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “…“human-likeness” or “engagingness”, either by automatically comparing with a human ground-truth reference, e.g., by using similarity metrics such as BLEURT (Sellam et al., 2020) or BERTscore (Zhang et al., 2020a), or by asking humans to evaluate this manually (Deriu et al., 2020; Li et al., 2019).”
- Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, methodology): “For example, Zhang et al. (2020a) trains a “side” network that is fused with the pretrained model via summation; adapter-tuning inserts task-specific layers (adapters) between each layer of the pretrained LM (Houlsby et al., 2019; Lin et al., 2020; Rebuffi et al., 2017; Pfeiffer et al., 2020).”
- CM3: A Causal Masked Multimodal Model of the Internet (2022, methodology): “Quantitatively we measure the quality of CM3 zero-shot captioning by evaluating using BERT-Score 2 (Zhang et al., 2019) with the RoBERTa-Large models (Liu et al., 2019) on the validation set from MS-COCO.”
- Recursively Summarizing Books with Human Feedback (2021, methodology): “SummaQA requires paragraph-aligned summaries, which we do not have, and so we report results on ROUGE and BERTScore.”
- On the Evaluation of Vision-and-Language Navigation Instructions (2021, methodology): “BERTScore (Zhang et al., 2019) takes a semi-model-based approach to compute token-wise similarity using contextual embeddings learned with BERT (Devlin et al.”
- Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models (2020, methodology): “More recently, embedding similarity based metrics such as HUSE (Shimanaka et al. 2018) and BERTScore (Zhang et al. 2019).”
- Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations (2021, methodology): “We only report automatic metrics (BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and BERTScore (Zhang et al., 2020)) for NLE quality for ComVE, since WinoGrande has only a small number of test NLE instances (which have been used for grounding in our human evaluation – see Appendix ”
- On Faithfulness and Factuality in Abstractive Summarization (2020, methodology): “Fourth, ROUGE (Lin and Hovy, 2003) and BERTScore (Zhang et al., 2020) correlates less with faithfulness/factuality than metrics derived from automatic semantic inference systems, specifically the degree to which a summary is entailed by the source document.”
- Survey of Hallucination in Natural Language Generation (2022, background): “Finally, BERTScore [229] measures the similarity between the generated answer and description, indicating faithfulness.”
- DART: Open-Domain Structured Data Record to Text Generation (2020, methodology): “The final score is then given by different intrinsic metrics, such as generalized precision, recall, and F-score used in BERTScore, and”
- GPTScore: Evaluate as You Desire (2023, methodology): “The models considered here include the 9 baseline models: ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, MoverScore, PRISM, BARTSCORE, BARTSCORE+CNN, and BARTSCORE+CNN+Para, and 19 GPTScore models built based on the GPT3-based, GPT2-based, OPT-based, and Flan-T5-based pre-trained models.”
- Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, background): “…BLEU (Papineni et al., 2002), NIST (Belz and Reiter, 2006), METEOR (Lavie and Agarwal, 2007), ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015), TER (Snover et al., 2005), MoverScore (Zhao et al., 2019), BERTScore (Zhang et al., 2019), BLEURT (Sellam et al., 2020) and PARENT (Dhingra et al., 2019”