Byte-Level Machine Reading Across Morphologically Varied Languages

By Tom Kenter, Llion Jones, Daniel Hewlett. 2018

16 citations recorded by the source corpus.

The machine reading task, where a computer reads a document and answers questions about it, is important in artificial intelligence research. Recently, many models have been proposed to address it. Word-level models, which have words as units of input and output, have proven to yield state-of-the-art results when evaluated on English datasets. However, in morphologically richer languages, many more unique words exist than in English due to highly productive prefix and suffix mechanisms. This may set back word-level models, since vocabulary sizes too big to allow for efficient computing may have to be employed. Multiple alternative input granularities have been proposed to avoid large input vocabularies, such as morphemes, character n-grams, and bytes. Bytes are advantageous as they provide a universal encoding format across languages, and allow for a small vocabulary size, which, moreover, is identical for every input language. In this work, we investigate whether bytes are suitable as input units across morphologically varied languages. To test this, we introduce two large-scale machine reading datasets in morphologically rich languages, Turkish and Russian. We implement 4 byte-level models, representing the major types of machine reading models and introduce a new seq2seq variant, called encoder-transformer-decoder. We show that, for all languages considered, there are models reading bytes outperforming the current state-of-the-art word-level baseline. Moreover, the newly introduced encoder-transformer-decoder performs best on the morphologically most involved dataset, Turkish. The large-scale Turkish and Russian machine reading datasets are released to public.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

1. Graham Neubig
2. Jiatao Gu
3. William Chan
4. Yu Zhang
5. Kyunghyun Cho
6. Changhan Wang
7. Yonghui Wu
8. Qun Liu

Citations in the corpus

• Character-Level Language Modeling with Deeper Self-Attention (2018, background): “Character-level modeling has shown promise in many areas such as sentiment analysis [35], question answering [21] and classification [50], and is an exciting area due to its simplicity and the ability to easily adapt to other languages.”
• Bytes Are All You Need: End-to-end Multilingual Speech Recognition and Synthesis with Bytes (2018, methodology): “[23] investigated the use of bytes for machine reading across morphologically varied languages and found them outperforming word-level baselines.”
• Training Multilingual Pre-trained Language Model with Byte-level Subwords (2021, methodology): “A vocabulary consisting purely of 256 bytes has also been used in many language and speech tasks such as part-of-speech tagging and named entity recognition [24], translation [25], machine reading [26] and speech recognition [27] but these work did not adopt BPE in text encoding.”
• CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network (2019, background): “…e.g., to encode sentence embeddings from the sequence of words in them (Serban et al., 2017; Kenter & de Rijke, 2017), or to construct sentence representation from a sequence of words, which are in turn treated as sequences of characters, or bytes (Jozefowicz et al., 2016; Kenter et al., 2018).”
• UvA-DARE (Digital Academic Repository) Learning to Transform, Combine, and Reason in Open-domain Question Answering (unknown): “Machine reading comprehension based models [4, 10, 13, 22, 24, 34] are the most used methods for extracting an answer in the question answering task.”
• Neural Machine Translation with Byte-Level Subwords (2019, methodology): “…bytes has previously been used in several natural language processing tasks: part-of-speech tagging and named entity recognition (Gillick et al. 2016), translation(Costa-Juss, Escolano, and Fonollosa 2017), machine reading (Kenter, Jones, and Hewlett 2018) and speech recognition (Li et al. 2019).”
• Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation (2021, background): “…clas-sification (Zhang et al., 2015; Zhang and LeCun, 2017), part-of-speech tagging and NER (Gillick et al., 2016; Akbik et al., 2018; Pinter et al., 2019), named entity detection (Yu et al., 2018), dependency parsing (Vania et al., 2018), and machine reading comprehension (Hewlett et al., 2018).”