Neural Machine Translation without Embeddings
By Uri Shaham, Omer Levy. 2020
17 citations recorded by the source corpus.
Many NLP models operate over sequences of subword tokens produced by hand-crafted tokenization rules and heuristic subword induction algorithms. A simple universal alternative is to represent every computerized text as a sequence of bytes via UTF-8, obviating the need for an embedding layer since there are fewer token types (256) than dimensions. Surprisingly, replacing the ubiquitous embedding layer with one-hot representations of each byte does not hurt performance; experiments on byte-to-byte machine translation from English to 10 different languages show a consistent improvement in BLEU, rivaling character-level and even standard subword-level models. A deeper investigation reveals that the combination of embeddingless models with decoder-input dropout amounts to token dropout, which benefits byte-to-byte models in particular.
Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Colin Raffel
- Yejin Choi
- Percy Liang
- Hannaneh Hajishirzi
- Jianfeng Gao
- Omer Levy
In Natural Language Processing
- Colin Raffel
- Omer Levy
- Noah Constant
- Rami Al-Rfou
- Hinrich Schütze
- Mohit Bansal
- Mihir Kale
- Hannaneh Hajishirzi
In Model Architectures
Citations in the corpus
- ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, background): “We show that ByT5 is competitive with mT5 on standard English and multilingual NLP benchmarks and outperforms mT5 at small model sizes.”
- How Optimal is Greedy Decoding for Extractive Question Answering? (2021, background): “It is quite likely that this phenomenon disappears when using character-level or byte-level tokenization (Shaham and Levy, 2021; Xue et al., 2021).”