Bridging the Gap for Tokenizer-Free Language Models
By Dokook Choe, Rami Al-Rfou, Mandy Guo, Heeyoung Lee, Noah Constant. 2019
27 citations recorded by the source corpus.
Purely character-based language models (LMs) have been lagging in quality on large scale datasets, and current state-of-the-art LMs rely on word tokenization. It has been assumed that injecting the prior knowledge of a tokenizer into the model is essential to achieving competitive results. In this paper, we show that contrary to this conventional wisdom, tokenizer-free LMs with sufficient capacity can achieve competitive performance on a large scale dataset. We train a vanilla transformer network with 40 self-attention layers on the One Billion Word (lm1b) benchmark and achieve a new state of the art for tokenizer-free LMs, pushing these models to be on par with their word-based counterparts.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- John Wieting
- Sabrina J. Mielke
- Colin Raffel
- J. Clark
- Chenglei Si
- Benoît Sagot
- Iulia Turc
- Zaid Alyafeai
Citations in the corpus
- Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation (2021, background): “Although the gap has been reduced (Choe et al., 2019), subword transformers remain the status quo for pure language modeling.”
- Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, background): “This finding was updated by Choe et al. (2019), who again manage to match previous word-and subword-based state-of-the-art language modeling results.”