Lightweight Adaptive Mixture of Neural and N-gram Language Models
By Anton Bakhtin, Arthur Szlam, Marc'Aurelio Ranzato, Edouard Grave. 2018
11 citations recorded by the source corpus.
It is often the case that the best performing language model is an ensemble of a neural language model with n-grams. In this work, we propose a method to improve how these two models are combined. By using a small network which predicts the mixture weight between the two models, we adapt their relative importance at each time step. Because the gating network is small, it trains quickly on small amounts of held out data, and does not add overhead at scoring time. Our experiments carried out on the One Billion Word benchmark show a significant improvement over the state of the art ensemble without retraining of the basic modules.
Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Michael Lewis
- Noah A. Smith
- Mohit Bansal
- Omer Levy
- Yejin Choi
- Percy Liang
- Hannaneh Hajishirzi
In Natural Language Processing
- Luke S. Zettlemoyer
- Michael Lewis
- Mohit Bansal
- Yejin Choi
- Omer Levy
- Percy Liang
- Graham Neubig
- Hannaneh Hajishirzi
In Model Architectures
- Michael Lewis
- Urvashi Khandelwal
- Luke S. Zettlemoyer
- Dan Jurafsky
- Omer Levy
- Yonatan Belinkov
- Ivan Titov
Citations in the corpus
- Generalization through Memorization: Nearest Neighbor Language Models (2019, methodology): “Figure 7 shows little improvement from using n -gram LMs – 0.2 perplexity points (similarly to Bakhtin et al. (2018)).”
- Gmail Smart Compose: Real-Time Assisted Writing (2019, neutral): “[2]).”
- Hash Layers For Large Sparse Models (2021, neutral): “Thus this work is related to previous work attempting to combine neural and n-gram language models [17, 18, 19, 20, 21, 22].”