A Hierarchical Neural Autoencoder for Paragraphs and Documents

By Jiwei Li, Minh-Thang Luong, Dan Jurafsky. 2015

614 citations recorded by the source corpus.

Natural language generation of coherent long texts like paragraphs or longer documents is a challenging problem for recurrent networks models. In this paper, we explore an important step toward this generation task: training an LSTM (Longshort term memory) auto-encoder to preserve and reconstruct multi-sentence paragraphs. We introduce an LSTM model that hierarchically builds an embedding for a paragraph from embeddings for sentences and words, then decodes this embedding to reconstruct the original paragraph. We evaluate the reconstructed paragraph using standard metrics like ROUGE and Entity Grid, showing that neural models are able to encode texts in a way that preserve syntactic, semantic, and discourse coherence. While only a first step toward generating coherent text units from neural models, our work has the potential to significantly impact natural language generation and summarization1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond (2016, methodology): “This model aims to capture this notion of two levels of importance by using two bi-directional RNNs on the source side, one at the word-level and the other at the sentence-level, similar to the model described in (Li et al., 2015).”
Generating Sentences from a Continuous Space (2015, background): “Sequence autoencoders have seen some success in pre-training sequence models for supervised downstream tasks (Dai and Le, 2015) and in generating complete documents (Li et al., 2015a).”
Hierarchical Neural Story Generation (2018, methodology): “For instance, Li et al. (2015b) use an LSTM to hierarchically learn word, then sentence, then paragraph embed-dings, then transform the paragraph embeddings into text.”
A Diversity-Promoting Objective Function for Neural Conversation Models (2015, background): “Direct decoding from (1 − λ ) log p ( T | S ) + λ log p ( S | T ) is intractable, as the second part (i.e., p ( S | T ) ) requires completion of target generation before p ( S | T ) can be effectively computed.”
Adversarial Learning for Neural Dialogue Generation (2017, methodology): “The input dialogue is encoded into a vector representation using a hierarchical encoder (Li et al., 2015; Serban et al., 2016b),2 which is then fed to a 2-class softmax function, returning the probability of the input dialogue episode being a machine-generated dialogue (denoted Q−({x, y})) or a huma”
Towards Universal Paraphrastic Sentence Embeddings (2015, methodology): “One approach is to train an autoencoder in an attempt to learn the latent structure of the sequence, whether it be a sentence with a parse tree (Socher et al., 2011), or a longer sequence such as a paragraph or document (Li et al., 2015b).”
Stacked Cross Attention for Image-Text Matching (2018, background): “Similarly, attention methods for natural language processing adaptively select and aggregate informative snippets to infer results [42, 35, 3, 25, 28].”
Dynamic Memory Networks for Visual and Textual Question Answering (2016, methodology): “Attention mechanisms have been successful components in both machine translation (Bahdanau et al., 2015; Luong module and memory module, to improve question answering.”
Hierarchical Attention Networks for Document Classification (2016, background): “There are some other works that use hierarchical structure in sequence generation (Li et al., 2015) and language modeling (Lin et al., 2015).”
Pay Attention to the Ending:Strong Neural Baselines for the ROC Story Cloze Task (2017, methodology): “When using our HIER model, we use a hierarchical recurrent neural network (RNN) (Li et al., 2015) with two levels.”
Dual Attention Networks for Multimodal Reasoning and Matching (2016, methodology): “This approach has been successfully applied to various tasks including machine translation [4], text generation [16], sentence summarization [25], and question answering [15, 32].”
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (2015, methodology): “We also take the last state of the recurrent layer I as a compact representation for the sentence, following the idea behind the Encoder-Decoder framework [9] in NMT.”
Document Modeling with Gated Recurrent Neural Network for Sentiment Classification (2015, background): “A recent work in (Li et al., 2015b) also investigate LSTM to model document meaning.”
Ranking Sentences for Extractive Summarization with Reinforcement Learning (2018, methodology): “Given a document D consisting of a sequence of sentences ( s 1 , s 2 ,... , s n ) , we follow common practice and feed sentences in reverse order (Sutskever et al., 2014; Li et al., 2015; Filippova et al., 2015; Narayan et al., 2017).”
ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs (2015, background): “…NLP domain after its success in computer vision and speech recognition, and mainly rely on recurrent neural network for end-to-end encoder-decoder sys-tem for tasks such as machine translation (Bahdanau et al., 2015; Luong et al., 2015a) and text reconstruction (Li et al., 2015; Rush et al., 2015).”
LCSTS: A Large Scale Chinese Short Text Summarization Dataset (2015, background): “Recently, recurrent neural network (RNN) have shown powerful abilities on speech recognition (Graves et al., 2013), machine translation (Sutskever et al., 2014) and automatic dialog response (Shang et al., 2015).”
Document Context Language Models (2015, methodology): “The value of L used in most experiments is 5 , although we compare with L = 10 in subsection 4.1.”