Gated Word-Character Recurrent Language Model

By Yasumasa Onoe, Kyunghyun Cho. 2016

106 citations recorded by the source corpus.

We introduce a recurrent neural network language model (RNN-LM) with long short-term memory (LSTM) units that utilizes both character-level and word-level inputs. Our model has a gate that adaptively finds the optimal mixture of the character-level and word-level inputs. The gate creates the final vector representation of a word by combining two distinct representations of the word. The character-level inputs are converted into vector representations of words using a bidirectional LSTM. The word-level inputs are projected into another high-dimensional space by a word lookup table. The final vector representations of words are used in the LSTM language model which predicts the next word given all the preceding words. Our model with the gating mechanism effectively utilizes the character-level inputs for rare and out-of-vocabulary words and outperforms word-level language models on several English corpora.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Using the Output Embedding to Improve Language Models (2016, methodology): “In addition to training our models on PTB and text8, following (Miyamoto and Cho, 2016), we also compare the performance of the NNLMs on the BBC (Greene and Cunningham, 2006) and IMDB (Maas et al., 2011) datasets, each of which we process and split into a train/validation/test split (we use the…”
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks (2016, background): “Recently, the joint use of word and character information has proven to be effective in language modeling (Miyamoto and Cho, 2016), but just using the simple character n -gram embeddings is fast and also effective.”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, background): “This is a highly studied but relatively small language modeling dataset (Miyamoto & Cho, 2016; Krueger et al., 2017; Mer-ity et al., 2017).”
Words or Characters? Fine-grained Gating for Reading Comprehension (2016, methodology): “Though Miyamoto & Cho (2016) also use a gate to choose between word-level and character-level representations, our method is different in two ways.”
Subword-augmented Embedding for Cloze Reading Comprehension (2018, background): “…tasks, character embedding has also beneﬁt other natural language process tasks, such as word segmentation (Cai et al., 2017), machine translation (Luong and Manning, 2016), tagging (Yang et al., 2016; Li et al., 2018) and language modeling (Verwimp et al., 2017; Miyamoto and Cho, 2016).”
Effective Subword Segmentation for Text Comprehension (2018, background): “Miyamoto and Cho (2016) introduced a recurrent neural network language model with LSTM units and a word-character gate to adaptively find the optimal mixture of the character-level and word-level inputs.”
Convolutional Sequence Modeling Revisited (2018, background): “This is a highly studied dataset in the field of language modeling (Miyamoto & Cho, 2016; Krueger et al., 2017; Merity et al., 2017), with exceptional results have been achieved by some highly optimized RNNs. Wikitext-103: Wikitext-103 (Merity et al., 2016) is almost 110 times as large as PTB,…”
Trellis Networks for Sequence Modeling (2018, background): “Though small, PTB has been a highly studied dataset in the domain of language modeling (Miyamoto & Cho, 2016; Zilly et al., 2017; Merity et al., 2018b; Melis et al., 2018; Yang et al., 2018).”
From Characters to Words to in Between: Do We Capture Morphology? (2017, methodology): “3 We use a common setup for all experiments based on that of Ling et al. (2015), Kim et al. (2016), and Miyamoto and Cho (2016).”
Effective Character-Augmented Word Embedding for Machine Reading Comprehension (2018, background): “Not only for machine reading comprehension tasks, character embedding has also benefited other natural language process tasks, such as word segmentation [2,3], machine translation [18,19], tagging [17,1,9] and language modeling [21,23].”
Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification (2019, methodology): “In this paper, we will refer to Miyamoto and Cho (2016)’s formula as simple-gated fusion and Lu et al. (2018)’s formula as symmetric-gated fusion .”
Visual Attention Model for Name Tagging in Multimodal Social Media (2018, methodology): “We propose to use a visual modulation gate, similar to (Miyamoto and Cho, 2016; Yang et al., 2016a), to dynamically control the combination of visual features and word representation generated by BLSTM at word-level, before feed them into the CRF layer for tag prediction.”