Soft Contextual Data Augmentation for Neural Machine Translation
By Jinhua Zhu, Fei Gao, Lijun Wu, Yingce Xia, Tao Qin, Wen-gang Zhou, Xueqi Cheng, Tie-Yan Liu. 2019
136 citations recorded by the source corpus.
While data augmentation is an important trick to boost the accuracy of deep learning methods in computer vision tasks, its study in natural language tasks is still very limited. In this paper, we present a novel data augmentation method for neural machine translation.Different from previous augmentation methods that randomly drop, swap or replace words with other words in a sentence, we softly augment a randomly chosen word in a sentence by its contextual mixture of multiple related words. More accurately, we replace the one-hot representation of a word by a distribution (provided by a language model) over the vocabulary, i.e., replacing the embedding of this word by a weighted combination of multiple semantically similar words. Since the weights of those words depend on the contextual information of the word to be replaced,the newly generated sentences capture much richer information than previous augmentation methods. Experimental results on both small scale and large scale machine translation data sets demonstrate the superiority of our method over strong baselines.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
In Natural Language Processing
Citations in the corpus
- Microsoft Research Asia’s Systems for WMT19 (2019, methodology): “SCA is a data augmentation technology for NMT (Zhu et al., 2019), which replaces a randomly chosen word in a sentence with its soft version.”
- A Survey of Data Augmentation Approaches for NLP (2021, methodology): “Gao et al. (2019) introduce S OFT C ONTEXTUAL DA that softly augments randomly chosen words in a sentence using a contextual mixture of multiple related words over the vocabulary.”
- Syntax-Aware Data Augmentation for Neural Machine Translation (2020, methodology): “DATA augmentation (DA) is a helpful strategy in deep learning to enhance accuracy and has been widely used in computer vision (CV) [1], natural language processing (NLP) [2], [3], [4], [5], [6], [7], [8], [9] and other areas.”
- Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “There have been subsequent attempts to select the best set of words considering luency [59, 189].”
- Substructure Substitution: Structured Data Augmentation for NLP (2021, methodology): “Data augmentation has been shown effective for various natural language processing (NLP) tasks, such as machine translation (Fadaee et al., 2017; Gao et al., 2019; Xia et al., 2019, inter alia ), text classification (Wei and Zou, 2019; Quteineh et al., 2020), semantic role labeling (Fürstenau and…”
- An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, unknown): “For example, instead of sampling one specific word from candidates by LMs, Gao et al. (2019) propose to compute a weighted average over embeddings of possible words predicted by LMs as the replaced input since the averaged representations could augment text with richer information.”
- An Analysis of Simple Data Augmentation for Named Entity Recognition (2020, background): “Gao et al. (2019) replace a randomly chosen word by a soft word, which is a probabilistic distribution over the vocabulary, provided by a language model.”
- A Diverse Data Augmentation Strategy for Low-Resource Neural Machine Translation (2020, methodology): “We compared our method with other data augmentation methods mentioned in Zhu’s work [26].”