Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization

By Faisal Ladhak, Esin Durmus, He He, Claire Cardie, K. McKeown. 2021

74 citations recorded by the source corpus.

Despite recent progress in abstractive summarization, systems still suffer from faithfulness errors. While prior work has proposed models that improve faithfulness, it is unclear whether the improvement comes from an increased level of extractiveness of the model outputs as one naive way to improve faithfulness is to make summarization models more extractive. In this work, we present a framework for evaluating the effective faithfulness of summarization systems, by generating a faithfulness-abstractiveness trade-off curve that serves as a control at different operating points on the abstractiveness spectrum. We then show that the Maximum Likelihood Estimation (MLE) baseline as well as recently proposed methods for improving faithfulness, fail to consistently improve over the control at the same level of abstractiveness. Finally, we learn a selector to identify the most faithful and abstractive summary for a given document, and show that this system can attain higher faithfulness scores in human evaluations while being more abstractive than the baseline system on two datasets. Moreover, we show that our system is able to achieve a better faithfulness-abstractiveness trade-off than the control at the same level of abstractiveness.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Measuring Attribution in Natural Language Generation Models (2021, background): “Possible model outputs fall on a spectrum ranging from synthesized information to the mostly unassailable extractive generations (Ladhak et al. 2022).”
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization (2022, methodology): “To this end, Ladhak et al. (2021) introduces a faithfulness-abstractiveness trade-off curve to measure the faithfulness given the model’s extractive-ness.”
Spurious Correlations in Reference-Free Evaluation of Text Generation (2022, background): “One simple measure of whether a generated summary is faithful is to look at its word overlap with the source article; summaries with a higher word overlap are more likely to be faithful (Ladhak et al., 2021).”
Evaluating Verifiability in Generative Search Engines (2023, background): “This inverse relationship between citation precision and perceived utility is symptomatic of a trade-off between faithfulness and abstractiveness (Ladhak et al., 2022).”
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations (2022, unknown): “Recent works (Dreyer et al., 2021; Ladhak et al., 2021) investigate the trade-off between factuality and abstractiveness of summaries and observe that factuality tends to drop with increased abstractiveness.”