Hierarchical Recurrent Neural Network for Document Modeling

By R. Lin, Shujie Liu, Muyun Yang, Mu Li, M. Zhou, Sheng Li. 2015

169 citations recorded by the source corpus.

This paper proposes a novel hierarchical recurrent neural network language model (HRNNLM) for document modeling. After establishing a RNN to capture the coherence between sentences in a document, HRNNLM integrates it as the sentence history information into the word level RNN to predict the word sequence with cross-sentence contextual information. A two-step training approach is designed, in which sentence-level and word-level language models are approximated for the convergence in a pipeline style. Examined by the standard sentence reordering scenario, HRNNLM is proved for its better accuracy in modeling the sentence coherence. And at the word level, experimental results also indicate a significant lower model perplexity, followed by a practical better translation result when applied to a Chinese-English document translation reranking task.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (2015, methodology): “We also take the last state of the recurrent layer I as a compact representation for the sentence, following the idea behind the Encoder-Decoder framework [9] in NMT.”
Document Context Language Models (2015, methodology): “Unlike some prior work on coherence evaluation (Li & Hovy, 2014; Li et al., 2015; Lin et al., 2015), our approach is not trained on supervised data.”
Learning Generic Sentence Representations Using Convolutional Neural Networks (2016, methodology): “our hierarchical CNN-LSTM model to extract meaningful sentence representations. However, our model can also be considered as a hierarchical language model for paragraphs. Similar work can be found in [20, 21, 22]. However, [20, 21] uses a LSTM for the sentence encoder, while [22] uses a bag-of-words”
TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency (2016, background): “Contextual RNN s have received a lot of attention (Mikolov and Zweig, 2012; Mikolov et al., 2014; Ji et al., 2015; Lin et al., 2015; Ji et al., 2016; Ghosh et al., 2016).”
Topic-based Evaluation for Conversational Bots (2018, background): “Deep learning advances have had a profound inﬂuence on natural language and dialog research, and researchers have experimented with Recurrent Neural Networks (RNNs)[11], [20], [13] and Convolutional Neural Networks (CNNs) [9], [4], [8] for text classiﬁcation and summarization tasks.”
Hierarchical Attention Networks for Document Classification (2016, background): “There are some other works that use hierarchical structure in sequence generation (Li et al., 2015) and language modeling (Lin et al., 2015).”
Message Passing Attention Networks for Document Understanding (2019, background): “. However, it might be beneﬁcial to explicitly capture document hierarchy. Hierarchical architectures have brought signiﬁcant improvements to many NLP tasks, such as language modeling and generation (Lin et al., 2015; Li et al., 2015), sentiment and topic classiﬁcation (Tang et al., 2015; Yang et al”
Weighted Neural Bag-of-n-grams Model: New Baselines for Text Classification (2016, methodology): “Denil et al. (2014), Lin et al. (2015), Li et al. (2015b) and Tang et al. (2015) respectively propose to use CNN-CNN, RNN-RNN, LSTM-LSTM and CNN-LSTM to represent documents hierarchically.”
Chunk-based Decoder for Neural Machine Translation (2017, methodology): “Hierarchical RNN s are used for various NLP tasks such as machine translation (Luong and Manning, 2016), document modeling (Li et al., 2015; Lin et al., 2015), dialog generation (Serban et al., 2017), image captioning (Krause et al., 2016), and video captioning (Yu et al., 2016).”
Unsupervised Learning of Sentence Representations using Convolutional Neural Networks (2016, methodology): “For reference, the CNN-LSTM composite model was trained for roughly two weeks.”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “more effectively process long-term context information and sequence information [79], [80], thereby avoiding the need”
A Latent Variable Recurrent Neural Network for Discourse-Driven Language Models (2016, background): “Wang and Cho (2015) and Lin et al. (2015) construct bag-of-words representations of previous sentences, which are then used to inform the RNN language model that generates the current sentence.”
Recurrent Topic-Transition GAN for Visual Paragraph Generation (2017, background): “To reason about long-term linguistic structures with multiple sentences, hierarchical recurrent network [19, 21, 35, 16] has been widely used to directly simulate the hierarchy of language.”
Distraction-Based Neural Networks for Modeling Document (2016, methodology): “This includes research that incorporates document-level information for language modeling [Wang and Cho, 2015; Lin et al., 2015] and that answers questions [Hermann et al.”
A Hierarchical Approach for Generating Descriptive Image Paragraphs (2016, background): “[19] use different recurrent units to model sentences and words.”
Multilingual Hierarchical Attention Networks for Document Classification (2017, result): “Lai et al. (2015) proposed a recurrent CNN to capture sequential information, which outperformed simpler CNNs. Lin et al. (2015) and Tang et al. (2015) proposed hierarchical recurrent NNs and showed that they were superior to CNN-based models.”
Contextual LSTM (CLSTM) models for Large scale NLP tasks (2016, methodology): “[28] use a hierarchical RNN structure for document-level as well as sentence-level modeling – they evaluate their models using word prediction perplexity, as well as an approach of coherence evaluation by trying to predict sentence-level ordering in a document.”
Sentence Ordering using Recurrent Neural Networks (2016, unknown): “More relevant to our work is a similar model (but without an encoder) considered by Lin et al. (2015).”