Multilingual Language Processing From Bytes

By D. Gillick, Clifford Brunk, Oriol Vinyals, A. Subramanya. 2015

229 citations recorded by the source corpus.

We describe an LSTM-based model which we call Byte-to-Span (BTS) that reads text as bytes and outputs span annotations of the form [start, length, label] where start positions, lengths, and labels are separate entries in our vocabulary. Because we operate directly on unicode bytes rather than language-specific words or characters, we can analyze text in many languages with a single model. Due to the small vocabulary size, these multilingual models are very compact, but produce results similar to or better than the state-of- the-art in Part-of-Speech tagging and Named Entity Recognition that use only the provided training datasets (no external data sources). Our models are learning "from scratch" in that they do not rely on any elements of the standard pipeline in Natural Language Processing (including tokenization), and thus can run in standalone fashion on raw text.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Language Modeling (2016, methodology): “This has shown to help when using byte-level input embed-dings for named entity recognition (Gillick et al., 2015), and we hope it will enable similar gains when used to map onto words.”
Sequence-Level Knowledge Distillation (2016, background): “, 2016), NER/POS-tagging (Gillick et al., 2016), image captioning (Vinyals et al.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Successfully representing thousands of languages might require character (Lee et al., 2017; Cherry et al., 2018), byte (Gillick et al., 2015) or even bit level modeling.”
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation (2016, background): “Multilingual language processing has also shown to be successful in domains other than translation (Gillick et al., 2016; Tsvetkov et al., 2016).”
Pointing the Unknown Words (2016, background): “The third category of the approaches changes the unit of input/output itself from words to a smaller resolution such as characters (Graves, 2013) or bytecodes (Sennrich et al., 2015; Gillick et al., 2015).”
Neural Architectures for Named Entity Recognition (2016, methodology): “Gillick et al. (2015) model the task of sequencelabeling as a sequence to sequence learning problem and incorporate character-based representations into their encoder model.”
A Character-level Decoder without Explicit Segmentation for Neural Machine Translation (2016, background): “Gillick et al. (2015) further showed that it is possible to train a recurrent neural network on unicode bytes, instead of characters or words, to perform part-of-speech tagging and named entity recognition.”
Fully Character-Level Neural Machine Translation without Explicit Segmentation (2016, background): “Also, we drew in-spiration for our multilingual models from previous work that showed the possibility of training a single recurrent model for multiple languages in domains other than translation (Tsvetkov et al., 2016; Gillick et al., 2015).”
Bytes Are All You Need: End-to-end Multilingual Speech Recognition and Synthesis with Bytes (2018, methodology): “In this work, we investigate the potential of representing text using byte sequences introduced in [22,23] for speech processing.”
From Characters to Words to in Between: Do We Capture Morphology? (2017, background): “…of characters (Ling et al., 2015; Kim et al., 2016), character n-grams (Sperr et al., 2013; Wieting et al., 2016; Bojanowski et al., 2016; Botha and Blun-som, 2014), bytes (Plank et al., 2016; Gillick et al., 2016), or combinations thereof (Santos and Zadrozny, 2014; Qiu et al., 2014).”
Neural Monkey: An Open-source Tool for Sequence Learning (2017, background): “Besides NMT, sequence-to-sequence (S2S) learning in general proved its usefulness in various other tasks including building a chatbot (Vinyals and Le, 2015), image captioning (Vinyals et al., 2015; Xu et al., 2015), or text segmentation and entity recognition (Gillick et al., 2016).”
Byte-based Neural Machine Translation (2017, methodology): “Gillick et al. (2016) propose an neural network that reads text as bytes and use this model in tasks of Part-of-Speech and Named Entity Recognition.”
Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss (2016, background): “These levels of representation were previously introduced in different efforts (Chrupała, 2013; Zhang et al., 2015; Ling et al., 2015; Santos and Zadrozny, 2014; Gillick et al., 2016; Kim et al., 2015), but a comparative evaluation was missing.”
Learning to generate one-sentence biographies from Wikidata (2017, methodology): “…another, other work been able to use these models by coercing different structures into sequences, e.g., ﬂattening trees for parsing (Vinyals et al., 2015), predicting span types and lengths over byte input (Gillick et al., 2016) or ﬂattening logical forms for semantic parsing (Xiao et al., 2016).”