Hierarchical Transformers for Multi-Document Summarization

By Yang Liu, Mirella Lapata. 2019

318 citations recorded by the source corpus.

In this paper, we develop a neural summarization model which can effectively process multiple input documents and distill Transformer architecture with the ability to encode documents in a hierarchical manner. We represent cross-document relationships via an attention mechanism which allows to share information as opposed to simply concatenating text spans and processing them as a flat sequence. Our model learns latent dependencies among textual units, but can also take advantage of explicit graph representations focusing on similarity or discourse relations. Empirical results on the WikiSum dataset demonstrate that the proposed architecture brings substantial improvements over several strong baselines.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “…that best-worst scaling (asking evaluators to choose the best and the worst items in a set) is an efficient and reliable method for collecting annotations, and this approach has been used to collect comparative evaluations of generated text (e.g., Liu and Lapata, 2019; Amplayo et al., 2021).”
Recursively Summarizing Books with Human Feedback (2021, methodology): “…use a hierarchical approach to generating ﬁnal summaries, either by having a hierarchical encoder (Cohan et al., 2018; Zhang et al., 2019c; Liu and Lapata, 2019a), or by ﬁrst running an extractive summarization model followed by an abstractive model (Subramanian et al., 2019; Liu et al.,…”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, background): “Recent progress in neural approaches to building semantically constrained text generation systems has shown tremendous improvements in this direction (Liu and Lapata, 2019; Guo et al., 2018; Durmus et al., 2020; Wang et al., 2020).”
Towards mental time travel: a hierarchical memory for reinforcement learning agents (2021, methodology): “Other recent works have used a hierarchy of transformers with different timescales for supervised tasks [30, 58, 55], for example encoding sentences with one transformer, and then using these embeddings as higher-level tokens.”
BP-Transformer: Modelling Long-Range Context via Binary Partitioning (2019, methodology): “(1) Hierarchical Transformers (Miculicich et al., 2018; Liu and Lapata, 2019) uses two Transformers in a hierarchical architecture: one Transformer models the sentence representation with word-level context, and another the document representation with the sentence-level context.”
A Survey of Transformers (2021, methodology): “Liu and Lapata [86] propose a similar hierarchical Transformer for multi-document summarization where the extracted low-level representations are aggregated using an attention layer with a global trainable query node and low-level representations as the source of key-value pairs.”
MSˆ2: Multi-Document Summarization of Medical Studies (2021, methodology): “, 2018), although some methods explicitly model multi-document structure using semantic graph approaches (Baumel et al., 2018; Liu and Lapata, 2019; Li et al., 2020).”
Controlling the Amount of Verbatim Copying in Abstractive Summarization (2019, methodology): “Following (Liu and Lapata 2019), we perform Best-Worst Scaling where a human selects the best and worst summary among six candidates.”
Data Augmentation for Abstractive Query-Focused Multi-Document Summarization (2021, methodology): “where, MHA is multi-head attention, MHP is multi-head pooling (Liu and Lapata 2019) which is applied on full query tokens (h), LN is layer normalization, and FFN is feedforward networks.”
Automatic Document Sketching: Generating Drafts from Analogous Texts (2021, background): “multiple input documents (Liu and Lapata, 2019; Chu and Liu, 2019), and multi-source machine translation (Nishimura et al.”
PRIMER: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization (2021, methodology): “…al., 2018; Li et al., 2020; Pasunuru et al., 2021), leveraging graph neural networks to connect information between the documents, or hierarchical (Liu and Lapata, 2019a; Fabbri et al., 2019; Jin et al., Figure 1: Prior work on pre-training mostly focuses on single document representation and…”
Understanding Points of Correspondence between Sentences for Abstractive Summarization (2020, background): “A number of recent efforts select important words and sentences from a given document, then let the summarizer attend to selected content to generate a summary (Gehrmann et al., 2018; Hsu et al., 2018; Chen and Bansal, 2018; Putra et al., 2018; Lebanoff et al., 2018; Liu and Lapata, 2019).”
Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding (2021, background): “Hierarchical ViT structures becomes popular both in vision (Liu et al. 2021; Vaswani et al. 2021) and NLP (Zhang, Wei, and Zhou 2019; Santra, Anusha, and Goyal 2021; Liu and Lapata 2019; Pappagari et al. 2019).”
Attention, please! A survey of neural attention models in deep learning (2021, background): “2019a), Hierarchical Transformer (Liu and Lapata 2019), HighWay Recurrent Transformer (Chiang et al.”
Multi-document Summarization via Deep Learning Techniques: A Survey (2020, methodology): “Transformer [74] ✓ ✓ ✓”
Heterogeneous Graph Neural Networks for Extractive Document Summarization (2020, methodology): “Recently, pre-trained language models are also applied in summarization for contextual word representations (Zhong et al., 2019a; Liu and Lapata, 2019b; Xu et al., 2019; Zhong et al., 2020).”