Robsut Wrod Reocginiton via Semi-Character Recurrent Neural Network

By Keisuke Sakaguchi, Kevin Duh, Matt Post, Benjamin Van Durme. 2016

98 citations recorded by the source corpus.

Language processing mechanism by humans is generally more robust than computers. The Cmabrigde Uinervtisy (Cambridge University) effect from the psycholinguistics literature has demonstrated such a robust word processing mechanism, where jumbled words (e.g. Cmabrigde / Cambridge) are recognized with little cost. On the other hand, computational models for word recognition (e.g. spelling checkers) perform poorly on data with such noise. Inspired by the findings from the Cmabrigde Uinervtisy effect, we propose a word recognition model based on a semi-character level recurrent neural network (scRNN). In our experiments, we demonstrate that scRNN has significantly more robust performance in word spelling correction (i.e. word recognition) compared to existing spelling checkers and character-based convolutional neural network. Furthermore, we demonstrate that the model is cognitively plausible by replicating a psycholinguistics experiment about human reading difficulty using our model.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Developmental and Educational Psychology, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Graham Neubig
3. Douwe Kiela
4. Adina Williams
5. Kyunghyun Cho
6. Joelle Pineau
7. Jianfeng Gao
8. Yonatan Belinkov

In Cognitive Neuroscience

1. Sumeet Singh
2. Joelle Pineau
3. Keita Kurita
4. Graham Neubig
5. Zachary Chase Lipton
6. Koustuv Sinha
7. Adina Williams

In Developmental and Educational Psychology

1. Joelle Pineau
2. Graham Neubig

In Natural Language Processing

1. Mohit Bansal
2. Adina Williams
3. Graham Neubig
4. Douwe Kiela
5. Koustuv Sinha
6. Joelle Pineau
7. Antonios Anastasopoulos
8. Prasanna Parthasarathi

In Model Architectures

1. Joelle Pineau
2. Graham Neubig
3. Zachary Chase Lipton
4. Prasanna Parthasarathi
5. Bhuwan Dhingra
6. Koustuv Sinha
7. Antonios Anastasopoulos
8. Danish Pruthi

Citations in the corpus

• Analysis Methods in Neural Language Processing: A Survey (2018, neutral): “For instance, Sanchez et al. (2018) and Glockner et al. (2018) extracted examples from SNLI (Bowman et al., 2015) and replaced specific words such as hypernyms, synonyms, and antonyms, followed by manual verification.”
• Combating Adversarial Misspellings with Robust Word Recognition (2019, methodology): “Our word recognition models build upon the RNN-based semi-character word recognition model due to Sakaguchi et al. (2017).”
• Towards Robust Toxic Content Classification (2019, methodology): “Sakaguchi et al. (2017) proposed the semi-character level recurrent neural network (scRNN) as a method of generating robust word representations.”
• Sometimes We Want Ungrammatical Translations (2021, background): “Yet some studies have also reported that NMT systems can be surprisingly brittle when presented with out-of-domain data (Luong and Manning, 2015), or when trained with noisy input data containing small orthographic (Sakaguchi et al., 2017; Belinkov and Bisk, 2018; Vaibhav et al., 2019; Niu et al., 2”
• Improving Robustness of Machine Translation with Synthetic Noise (2019, background): “Machine Translation (MT) systems have been shown to exhibit severely degraded performance when presented with translation of out-of-domain or noisy data (Luong and Manning, 2015; Sakaguchi et al., 2016; Belinkov and Bisk, 2017).”