Submodular Optimization-based Diverse Paraphrasing and its Effectiveness in Data Augmentation

By Ashutosh Kumar, Satwik Bhattamishra, Manik Bhandari, Partha P. Talukdar. 2019

128 citations recorded by the source corpus.

Inducing diversity in the task of paraphrasing is an important problem in NLP with applications in data augmentation and conversational agents. Previous paraphrasing approaches have mainly focused on the issue of generating semantically similar paraphrases while paying little attention towards diversity. In fact, most of the methods rely solely on top-k beam search sequences to obtain a set of paraphrases. The resulting set, however, contains many structurally similar sentences. In this work, we focus on the task of obtaining highly diverse paraphrases while not compromising on paraphrasing quality. We provide a novel formulation of the problem in terms of monotone submodular function maximization, specifically targeted towards the task of paraphrasing. Additionally, we demonstrate the effectiveness of our method for data augmentation on multiple tasks such as intent classification and paraphrase recognition. In order to drive further research, we have made the source code available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

A Survey of Data Augmentation Approaches for NLP (2021, methodology): “, 2020) × 3 Depends Input 3 DIPS (Kumar et al., 2019a) × 3 tok Input 3 AUGMENTED SBERT (Thakur et al.”
Do Not Have Enough Data? Deep Learning to the Rescue! (2019, methodology): “The approaches include using variational autoencoding (VAE) (Kingma and Welling 2014), round-trip translation (Yu et al. 2018), paraphrasing (Kumar et al. 2019), and methods based on generative adversarial networks (Tanaka and Aranha 2019).”
MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification (2020, background): “Similarly, Kumar et al. (2019) proposed a new para-phrasing formulation in terms of monotone sub-modular function maximization to obtain highly diverse paraphrases, and Xie et al. (2019) and Chen et al. (2020) applied back translations (Sennrich et al., 2015) and word replacement to generate…”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, unknown): “Model Generation [18], [69], [105], [106], [107] [18], [108], [109], [110] [18], [16], [68], [17], [111], [76]”
Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models (2020, methodology): “…supervised baselines: the pointer generator trained by imitation learning (PG-IL) as in Du and Ji (2019), the diversity-promoting DiPS model (Kumar et al., 2019), and a finetuned BART model (Lewis et al., 2019), which uses a more complex pretraining method than our LMs. Note that DiPS…”
Syntax-Guided Controlled Generation of Paraphrases (2020, background): “…1983; Barzilay and Lee, 2003; Quirk et al., 2004; Hassan et al., 2007; Zhao et al., 2008; Madnani and Dorr, 2010; Wubben et al., 2010), they have recently been superseded by deep learning-based approaches (Prakash et al., 2016; Gupta et al., 2018; Li et al., 2019, 2018; Kumar et al., 2019).”
Reformulating Unsupervised Style Transfer as Paraphrase Generation (2020, unknown): “…generation is often encouraged during inference time via heuristic modiﬁcations to beam search (Li et al., 2016; Vijayakumar et al., 2018), nucleus sampling (Holtzman et al., 2020) or submodular optimization (Kumar et al., 2019); in contrast, we simply ﬁlter our training data to increase diversity.”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, methodology): “Paraphrasing has been widely adopted as a data augmentation technique in various NLP tasks (Yu et al., 2018; Xie et al., 2019; Kumar et al., 2019; He et al., 2019; Chen et al., 2020b,c; Cai et al., 2020), as it generally provides more diverse augmented text with different word choices and sentence s”
NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation (2021, background): “Previous work has identified discourse markers that have low ambiguity (Pitler et al., 2008).”
CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding (2020, methodology): “Previous efforts in augmenting text data were mainly focused on answering the first question (Yu et al., 2018; Xie et al., 2019; Kumar et al., 2019; Wei & Zou, 2019; Chen et al., 2020a; Shen et al., 2020).”
Paraphrase Generation: A Survey of the State of the Art (2021, methodology): “And it is also a good way for data augmentation (Kumar et al., 2019; Gao et al., 2020).”