Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model

By Sabrina J. Mielke, Jason Eisner. 2018

34 citations recorded by the source corpus.

We show how the spellings of known words can help us deal with unknown words in open-vocabulary NLP tasks. The method we propose can be used to extend any closedvocabulary generative model, but in this paper we specifically consider the case of neural language modeling. Our Bayesian generative story combines a standard RNN language model (generating the word tokens in each sentence) with an RNNbased spelling model (generating the letters in each word type). These two RNNs respectively capture sentence structure and word structure, and are kept separate as in linguistics. By invoking the second RNN to generate spellings for novel words in context, we obtain an open-vocabulary language model. For known words, embeddings are naturally inferred by combining evidence from type spelling and token context. Comparing to baselines (including a novel strong baseline), we beat previous work and establish state-of-the-art results on multiple datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology (2019, methodology): “For each language, we trained the BPE-RNNLM baseline open-vocabulary language model of Mielke and Eisner (2018) using the original corpus, the corpus following CDA using naı̈ve swapping of gendered words, and the corpus following CDA using our approach.”
An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (2019, methodology): “Additionally, we aim at exploring ways for scaling our approach to larger vocabulary sizes (Kumar and Tsvetkov, 2019) and for better handling of out-of-vocabulary words (OOV) (Mielke and Eisner, 2018; Sennrich et al., 2015) in order to be applicable to diverse datasets.”
Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation (2021, unknown): “, 2017) or used a character-only generator as a fallback when the main closedvocabulary word generator produced a rare or unknown token (Matthews et al., 2019; Mielke and Eisner, 2019).”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, background): “…information While word-level models are conceptually easy to understand and in the neural era (Bengio et al., 2001; Collobert and Weston, 2008; Mikolov et al., 2013) offer features at an interpretable granularity, their central weakness is the inability to deal with rare and novel words,…”
Maybe Deep Neural Networks are the Best Choice for Modeling Source Code (2019, unknown): “Simultaneously with our work, [52] independently developed a recent LM based on subword units for natural language, and found that it had close to state-of-the-art performance.”