SUM-QE: a BERT-based Summary Quality Estimation Model

By Stratos Xenouleas, Prodromos Malakasiotis, Marianna Apidianaki, Ion Androutsopoulos. 2019

39 citations recorded by the source corpus.

We propose SUM-QE, a novel Quality Estimation model for summarization based on BERT. The model addresses linguistic quality aspects that are only indirectly captured by content-based approaches to summary evaluation, without involving comparison with human references. SUM-QE achieves very high correlations with human ratings, outperforming simpler models addressing these linguistic aspects. Predictions of the SUM-QE model can be used for system development, and to inform users of the quality of automatically produced summaries and other types of generated text.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Caiming Xiong
2. Richard Socher
3. Mohit Bansal
4. Luke S. Zettlemoyer
5. Percy Liang
6. Noah A. Smith
7. Yejin Choi
8. Nitish Shirish Keskar

In Natural Language Processing

1. Caiming Xiong
2. Richard Socher
3. Bryan McCann
4. Nitish Shirish Keskar
5. Mohit Bansal
6. Thibault Sellam
7. Luke S. Zettlemoyer
8. Graham Neubig

In Evaluation & Benchmarks

1. Thibault Sellam
2. Caiming Xiong
3. Ankur P. Parikh
4. John Bohannon
5. Oleg V. Vasilyev
6. Dipanjan Das
7. Elizabeth Clark
8. Bryan McCann

Citations in the corpus

• CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “Yet language models show potential for mitigating these problems directly (Trinh & Le, 2018; Radford et al., 2019) or indirectly (Ra-jani et al., 2019; Xenouleas et al., 2019; Scialom et al., 2019).”
• BLEURT: Learning Robust Metrics for Text Generation (2020, methodology): “Sum-QE (Xenouleas et al., 2019) fine-tunes BERT for quality estimation as we describe in Section 3.”
• Finding a Balanced Degree of Automation for Summary Evaluation (2021, unknown): “Besides, some reference-free metrics are introduced for summary quality estimation (Xenouleas et al., 2019; Gao et al., 2020; Vasilyev et al., 2020) or faithfulness evaluation (Durmus et al., 2020; Wang et al., 2020).”
• Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluation (2021, background): “While human assessment can be favorably biased toward more extractive summaries (Ziegler et al., 2020), this bias can be mitigated by splitting the overall score into multiple dimensions (Xenouleas et al., 2019; Kryscinski et al., 2019, 2020; Fan et al., 2018; Zhu et al., 2020).”
• Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE? (2022, background): “Researchers also attempted to develop meth-ods for evaluating reference-free model summaries (Louis and Nenkova, 2013; Xenouleas et al., 2019).”
• MathBERT: A Pre-Trained Model for Mathematical Formula Understanding (2021, methodology): “These pre-trained models performed well in general NLP tasks like text classification [Devlin et al. , 2018], machine translation [Zhu et al. , 2020; Sundararaman et al. , 2019] and machine summarization [Miller, 2019; Xenouleas et al. , 2019].”
• Fill in the BLANC: Human-free quality estimation of document summaries (2020, background): “quality scores (Louis and Nenkova, 2009, 2013; Xenouleas et al., 2019).”