BRIO: Bringing Order to Abstractive Summarization

By Yixin Liu, Pengfei Liu, Dragomir R. Radev, Graham Neubig. Annual Meeting of the Association for Computational Linguistics, 2022

333 citations recorded by the source corpus.

Abstractive summarization models are commonly trained using maximum likelihood estimation, which assumes a deterministic (one-point) target distribution in which an ideal model will assign all the probability mass to the reference summary. This assumption may lead to performance degradation during inference, where the model needs to compare several system-generated (candidate) summaries that have deviated from the reference summary. To address this problem, we propose a novel training paradigm which assumes a non-deterministic distribution so that different candidate summaries are assigned probability mass according to their quality. Our method achieves a new state-of-the-art result on the CNN/DailyMail (47.78 ROUGE-1) and XSum (49.07 ROUGE-1) datasets. Further analysis also shows that our model can estimate probabilities of candidate summaries that are more correlated with their level of quality.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/58c9838b45edad4b7fc11bd7119f113ea5a3d7f1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

SLiC-HF: Sequence Likelihood Calibration with Human Feedback (2023, background): “Recently, another class of sequence-level contrastive methods [12, 21] seek to align model likelihood with an arbitrary, possibly non-differentiable reward, presenting an alternative to RL for optimizing the expected reward of samples.”
RankGen: Improving Text Generation with Large Ranking Models (2022, methodology): “…et al., 2022; An et al., 2022), use BERT for sentence-level language modeling (Ip-polito et al., 2020), and designing sequence-level losses (Wiseman and Rush, 2016; Edunov et al., 2018; Welleck et al., 2020; Liu et al., 2022) for reducing exposure bias (Bengio et al., 2015; Ranzato et al., 2016).”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “BRIO-Ext (Liu et al., 2022) uses BRIO’s reranker on candidate extractive summaries from MatchSum.”
Benchmarking Large Language Models for News Summarization (2023, methodology): “To contextualize the performance of LLMs, we mainly compare to previous state-of-the-art approaches that leveraged supervised finetuning (Liu and Lapata, 2019; Lewis et al., 2019; Zhang et al., 2020; Liu et al., 2022b).”
Aligning Large Language Models with Human: A Survey (2023, methodology): “Motivated by these ﬁndings, Liu et al. (2022b) recommend training LMs to produce edit operations between source (i.e., low-quality responses) and target (i.e., high-quality responses) sequences, which are subsequently integrated into a dynamic programming framework.”
News Summarization and Evaluation in the Era of GPT-3 (2022, methodology): “BRIO (Liu et al., 2022), a fine-tuned summarization model that reports state-of-the art results on both CNN/DM and XSum .”