Neural Lattice Language Models

By Jacob Buckman, Graham Neubig. 2018

30 citations recorded by the source corpus.

In this work, we propose a new language modeling paradigm that has the ability to perform both prediction and moderation of information flow at multiple granularities: neural lattice language models. These models construct a lattice of possible paths through a sentence and marginalize across this lattice to calculate sequence probabilities or optimize parameters. This approach allows us to seamlessly incorporate linguistic intuitions — including polysemy and the existence of multiword lexical items — into our language model. Experiments on multiple language modeling tasks show that English neural lattice language models that utilize polysemous embeddings are able to improve perplexity by 9.95% relative to a word-level baseline, and that a Chinese model that handles multi-character tokens is able to improve perplexity by 20.94% relative to a character-level baseline.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Adaptive Input Representations for Neural Language Modeling (2018, background): “There are different choices for the basic unit we wish to model, including full words (Bengio et al., 2003), characters for the input (Kim et al., 2016), or also the output (Merity et al., 2018) as well as sub-words (Buckman & Neubig, 2018; Mielke & Eisner, 2018).”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, background): “Buckman and Neubig (2018) tokeniza-tions that produced a low downstream loss, and the latter using just one tokenization sampled from the conditioned (and tempered) LM.”
Latent Relation Language Models (2019, background): “, 2018), chunking of word sequences into n-grams (Buckman and Neubig, 2018), deciding between input sources (Ling et al.”
Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation (2021, background): “Especially after the popularization of the inherently ambiguous subword vocabularies like BPE, several studies removed the assumption of a single correct input segmentation and marginalized over all possible segmentations (Van et al., 2017; Buckman and Neubig, 2018; Grave et al., 2019).”