SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation

By Xinyi Wang, Hieu Pham, Zihang Dai, Graham Neubig. 2018

210 citations recorded by the source corpus.

In this work, we examine methods for data augmentation for text-based tasks such as neural machine translation (NMT). We formulate the design of a data augmentation policy with desirable properties as an optimization problem, and derive a generic analytic solution. This solution not only subsumes some existing augmentation schemes, but also leads to an extremely simple data augmentation strategy for NMT: randomly replacing words in both the source sentence and the target sentence with other random words from their corresponding vocabularies. We name this method SwitchOut. Experiments on three translation datasets of different scales show that SwitchOut yields consistent improvements of about 0.5 BLEU, achieving better or comparable performances to strong alternatives such as word dropout (Sennrich et al., 2016a). Code to implement this method is included in the appendix.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Unsupervised Data Augmentation for Consistency Training (2019, methodology): “For Yelp-5, apart from back-translation, we include a simpler method Switchout [25] which replaces a token with a random”
Grammar-Based Grounded Lexicon Learning (2022, neutral): “…span recombination rules; (4) WordDrop [17] performs random dropout for input sequence (while keeping the same label); (5) similarly, SwitchOut [45] randomly replaces an input token with a random token from the vocabulary; (6) SeqMix [17] uses soft augmentation techniques following [49], which…”
Sequence-to-Sequence Speech Recognition with Time-Depth Separable Convolutions (2019, background): “Instead, we propose random sampling, where the previous prediction is replaced with a randomly sampled token [22].”
Automatically Learning Data Augmentation Policies for Dialogue Tasks (2019, neutral): “…Lemley et al., 2017) and NLP (F¨urstenau and Lapata, 2009; Sen-nrich et al., 2016; Wang and Yang, 2015; Zhang et al., 2015; Jia and Liang, 2016; Kim and Rush, 2016; Hu et al., 2017; Xu et al., 2017; Xia et al., 2017; Silfverberg et al., 2017; Kaﬂe et al., 2017; Hou et al., 2018; Wang et al., 2018).”
Optimal Completion Distillation for Sequence Learning (2018, background): “, 2016) and its variants (Ma et al., 2017; Elbayad et al., 2018; Wang et al., 2018) are also similiar to RL-based approaches.”
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout (2019, methodology): “, 2016), a popular semi-supervised learning method, has been well studied in neural machine translation (Hoang et al., 2018; Wang et al., 2018b; Edunov et al., 2018; Prabhumoye et al., 2018).”
Text Augmentation in a Multi-Task View (2021, background): “Despite this uncertainty, many augmentation techniques in NLP still generate examples stochastically and ignore original data (Zhang et al., 2015; Sennrich et al., 2016a; Xie et al., 2017; Li et al., 2017; Kobayashi, 2018; Wang et al., 2018).”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, methodology): “[70] propose a method that randomly replaces words in the input and target sentences with other words in the vocabulary.”
Syntax-Aware Data Augmentation for Neural Machine Translation (2020, methodology): “We also compare our methods with SwitchOut [40], WordDropout [43] and three original data augmentation methods Blanking, Dropout and Replace-”
Data Diversification: A Simple Strategy For Neural Machine Translation (2019, background): “[30] suggested to simply replace random words with other words in the vocabularies.”
Neural Machine Translation: A Review (2019, methodology): “via subword regularization [259], SwitchOut [283], or noisy back-translation [282] (see”
AdvAug: Robust Adversarial Augmentation for Neural Machine Translation (2020, methodology): “Existing methods in NMT may be divided into two categories, based upon extra corpora (Sennrich et al., 2016a; Cheng et al., 2016; Zhang and Zong, 2016; Edunov et al., 2018) or original parallel corpora (Fadaee et al., 2017; Wang et al., 2018; Cheng et al., 2019).”
Multi-Source Neural Machine Translation with Missing Data (2018, methodology): “proposed a method of randomly replacing words in both the source sentence and the target sentence with other random words from their corresponding vocabularies [20].”
Multi-Source Neural Machine Translation with Data Augmentation (2018, methodology): “proposed a method of randomly replacing words in both the source sentence and the target sentence with other random words from their corresponding vocabularies [16].”
SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness (2020, methodology): “SwitchOut (Wang et al., 2018) applies a noise function similar to RAML to both the source and target side.”
Soft Contextual Data Augmentation for Neural Machine Translation (2019, methodology): “Existing augmentation methods generate new training samples by replacing one word in the original sentences with another word (Wang et al., 2018; Kobayashi, 2018; Xie et al., 2017; Fadaee et al., 2017).”