HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization

By Xingxing Zhang, Furu Wei, M. Zhou. 2019

420 citations recorded by the source corpus.

Neural extractive summarization models usually employ a hierarchical encoder for document encoding and they are trained using sentence-level labels, which are created heuristically using rule-based methods. Training the hierarchical encoder with these inaccurate labels is challenging. Inspired by the recent work on pre-training transformer sentence encoders (Devlin et al., 2018), we propose Hibert (as shorthand for HIerachical Bidirectional Encoder Representations from Transformers) for document encoding and a method to pre-train it using unlabeled data. We apply the pre-trained Hibert to our summarization model and it outperforms its randomly initialized counterpart by 1.25 ROUGE on the CNN/Dailymail dataset and by 2.0 ROUGE on a version of New York Times dataset. We also achieve the state-of-the-art performance on these two datasets.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Recursively Summarizing Books with Human Feedback (2021, methodology): “SummaQA requires paragraph-aligned summaries, which we do not have, and so we report results on ROUGE and BERTScore.”
Text Summarization with Pretrained Encoders (2019, background): “Pretrained language models (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Dong et al., 2019; Zhang et al., 2019) have recently emerged as a key technology for achieving impressive gains in a wide variety of natural language tasks.”
ETC: Encoding Long and Structured Inputs in Transformers (2020, methodology): “Many approaches have been proposed to address this, like hierarchical processing (Zhang et al., 2019), sparse attention (Child et al., 2019), and segment-level recurrence (Dai et al., 2019).”
Pre-trained models for natural language processing: A survey (2020, background): “[206] tries to pre-trained a document-level model that predicts sentences instead of words, and then apply it on downstream tasks such as summarization.”
ETC: Encoding Long and Structured Data in Transformers (2020, methodology): “Many extensions of the original attention mechanism have been proposed recently to address this issue, including hierarchical processing of inputs (Zhang et al., 2019), sparse attention (Child et al.”
Towards mental time travel: a hierarchical memory for reinforcement learning agents (2021, methodology): “Other works have used a hierarchy of transformers with different timescales for supervised tasks [33, 63, 60, 31], for example encoding sentences with one transformer, and then using these embeddings as higher-level tokens.”
ReadTwice: Reading Very Large Documents with Memories (2021, neutral): “Another strategy such as HIBERT (Zhang et al., 2019) splits inputs into smaller segments which are processed individually, then assembled into a hierarchical representation.”
Cooperative Generator-Discriminator Networks for Abstractive Summarization with Narrative Flow (2019, methodology): “Following (Zhang et al., 2019a), we then compute precision, recall and F1 scores from these contextual representations.”
Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding (2021, methodology): “Hierarchical ViT structures becomes popular both in vision (Liu et al. 2021; Vaswani et al. 2021) and NLP (Zhang, Wei, and Zhou 2019; Santra, Anusha, and Goyal 2021; Liu and Lapata 2019; Pappagari et al. 2019).”
A Survey of Transformers (2021, methodology): “For document summarization, HIBERT [166] encodes a document of text by first learn sentence representations for all sentences and then use these sentence representations to encode document-level representations that are then used to generate the summary.”
Re-evaluating Evaluation in Text Summarization (2020, methodology): “2, finding that: (1) MoverScore and JS-2, which were the best performing metrics on TAC, have poor correlations with humans in comparing CNNDM Ext systems.”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “[214]; Capturing Critical Semantics: Nguyen et al.”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, background): “A very influential recent work, [208], introduces HIBERT ( HIerachical Bidirectional Encoder Representations from Transformers) as a pre-trained initialization for document summarization and report state-ofthe-art performance.”
Transforming Wikipedia Into Augmented Data for Query-Focused Summarization (2019, background): “Kulkarni et al. [36] propose SIBERT that extends HIBERT [37] to query-focused multi-document summarization by introducing a cross-document infusion layer and incorporating queries as additional contexts.”
Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries (2021, background): “Pre-trained language models have achieved promising results in abstractive text summarization (Edunov et al., 2019; Dong et al., 2019; Song et al., 2019; Zhang et al., 2019, 2020).”