Topically Driven Neural Language Model
By Jey Han Lau, Timothy Baldwin, Trevor Cohn. 2017
68 citations recorded by the source corpus.
Language models are typically applied at the sentence level, without access to the broader document context. We present a neural language model that incorporates document context in the form of a topic model-like architecture, thus providing a succinct representation of the broader document context outside of the current sentence. Experiments over a range of datasets demonstrate that our model outperforms a pure sentence-based model in terms of language model perplexity, and leads to topics that are potentially more coherent than those produced by a standard LDA topic model. Our model also has the ability to generate related sentences for a topic, providing another way to interpret topics.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Zhe Gan
- Lawrence Carin
- Mohit Bansal
- Jianfeng Gao
- Chunyuan Li
- Percy Liang
- Dinghan Shen
- Luke S. Zettlemoyer
Citations in the corpus
- Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context (2018, background): “Differences in nearby vs. long-range context suggest that memory models, which feed explicit context representations to the LSTM (Ghosh et al., 2016; Lau et al., 2017), could benefit from representations that specifically capture information orthogonal to that modeled by the LSTM.”
- On-The-Fly Information Retrieval Augmentation for Language Models (2020, background): “Various language models have utilized external knowledge or long contexts (Paperno et al., 2016; Yang and Mitchell, 2017; Peng et al., 2019; Khandelwal et al., 2018; Ghosh et al., 2016; Lau et al., 2017; Grave et al., 2016; Parthasarathi and Pineau, 2018).”
- Topic Compositional Neural Language Model (2017, methodology): “, either through concatenating the topic vector as an additional feature of RNNs (Mikolov and Zweig, 2012; Lau et al., 2017), or re-scoring the predicted distribution over words using the topic vector (Dieng et al.”