Incorporating Side Information into Recurrent Neural Network Language Models

By Cong Duy Vu Hoang, Trevor Cohn, Gholamreza Haffari. 2016

35 citations recorded by the source corpus.

Recurrent neural network language models (RNNLM) have recently demonstrated vast potential in modelling long-term dependencies for NLP problems, ranging from speech recognition to machine translation. In this work, we propose methods for conditioning RNNLMs on external side information, e.g., metadata such as keywords, description, document title or topic headline. Our experiments show consistent improvements of RNNLMs using side information over the baselines for two different datasets and genres in two languages. Interestingly, we found that side information in a foreign language can be highly beneficial in modelling texts in another language, serving as a form of cross-lingual language modelling.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Mohit Bansal
3. Percy Liang
4. Jason Aaron Edward Weston
5. Luke S. Zettlemoyer
6. Wen-tau Yih
7. Graham Neubig
8. Noah A. Smith

In Large Language Models

1. Jianfeng Gao
2. Wen-tau Yih
3. Ruslan Salakhutdinov
4. Marjan Ghazvininejad
5. Michel Galley
6. Yao-Hung Hubert Tsai
7. Ming-Wei Chang
8. Chris Brockett

In Natural Language Processing

1. Jianfeng Gao
2. Mohit Bansal
3. Luke S. Zettlemoyer
4. Jason Aaron Edward Weston
5. Wen-tau Yih
6. Yejin Choi
7. Graham Neubig
8. Michel Galley

Citations in the corpus

• A Knowledge-Grounded Neural Conversation Model (2017, methodology): “The dialog encoder and response decoder form together a sequence-to-sequence (S EQ 2S EQ model (Hochreiter and Schmidhuber, 1997; Sutskever et al., 2014), which has been sucessfully used in building end-to-end conversational systems (Sor-doni et al., 2015; Vinyals and Le, 2015; Li et al., 2016a).”
• Controlling Linguistic Style Aspects in Neural Language Generation (2017, neutral): “1 See (Hoang et al., 2016) for other conditioning models.”
• Deep Neural Networks for Bot Detection (2018, methodology): “Related architectures that use side-information to enhance recurrent model representations have been alluded to by some authors working primarily in language models [4, 29, 41], but has never been used in the context of social media classification to the best of our knowledge.”
• Improving One-Shot Learning through Fusing Side Information (2017, background): “Hoang et al. (2016) proposed to condition the recurrent neural network language models on metadata, such as document titles, authorship, and time stamp.”