Focus Attention: Promoting Faithfulness and Diversity in Summarization

By Rahul Aralikatte, Shashi Narayan, Joshua Maynez, Sascha Rothe, Ryan T. McDonald. 2021

53 citations recorded by the source corpus.

Professional summaries are written with document-level information, such as the theme of the document, in mind. This is in contrast with most seq2seq decoders which simultaneously learn to focus on salient content, while deciding what to generate, at each decoding step. With the motivation to narrow this gap, we introduce Focus Attention Mechanism, a simple yet effective method to encourage decoders to proactively generate tokens that are similar or topical to the input document. Further, we propose a Focus Sampling method to enable generation of diverse summaries, an area currently understudied in summarization. When evaluated on the BBC extreme summarization task, two state-of-the-art models augmented with Focus Attention generate summaries that are closer to the target and more faithful to their input documents, outperforming their vanilla counterparts on ROUGE and multiple faithfulness measures. We also empirically demonstrate that Focus Sampling is more effective in generating diverse and faithful summaries than top-k or nucleus sampling-based decoding methods.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…more human ratings, and introduce initialization and and pretraining schemes (Sellam et al., 2020; Rei et al., 2020; Pu et al., 2021; Wegmann and Nguyen, 2021, inter alia), or focus on specific aspects like the faithfulness of generated text (e.g., Kryscinski et al., 2020; Aralikatte et al., 2021).”
Survey of Hallucination in Natural Language Generation (2022, background): “[3] introduce a short circuit from the input document to the vocabulary distribution via source-conditioned bias.”
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021, background): “The second type targets at modifying the sequence-to-sequence architecture to incorporate relation triplets (Cao et al., 2018), knowledge graphs (Zhu et al., 2021), and topics (Aralikatte et al., 2021) to inform the sum-marizers of article facts.”