Tailor: Generating and Perturbing Text with Semantic Controls

By Alexis Ross, Tongshuang Sherry Wu, Hao Peng, Matthew E. Peters, Matt Gardner. 2021

88 citations recorded by the source corpus.

Controlled text perturbation is useful for evaluating and improving model generalizability. However, current techniques rely on training a model for every target perturbation, which is expensive and hard to generalize. We present Tailor, a semantically-controlled text generation system. Tailor builds on a pretrained seq2seq model and produces textual outputs conditioned on control codes derived from semantic representations. We craft a set of operations to modify the control codes, which in turn steer generation towards targeted attributes. These operations can be further composed into higher-level ones, allowing for flexible perturbation strategies. We demonstrate the effectiveness of these perturbations in multiple applications. First, we use Tailor to automatically create high-quality contrast sets for four distinct natural language processing (NLP) tasks. These contrast sets contain fewer spurious artifacts and are complementary to manually annotated ones in their lexical diversity. Second, we show that Tailor perturbations can improve model generalization through data augmentation. Perturbing just ∼2% of training data leads to a 5.8-point gain on an NLI challenge set measuring reliance on syntactic heuristics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Red Teaming Language Models with Language Models (2022, neutral): “Other work uses LMs to aid crowdworkers in writing examples (Wu et al., 2021b; Ross et al., 2021; Bartolo et al., 2021b), a promising setting where our approach can also be used.”
Discovering Language Model Behaviors with Model-Written Evaluations (2022, methodology): “Hybrid approaches have succeeded in generating evaluation data (Wu et al., 2021; Ross et al., 2021), training data (Liu et al.”
Adversarial Training for High-Stakes Reliability (2022, background): “Adversarial training for language models There is a large and growing literature on both adversarial attacks and adversarial training for large language models [32, 33, 34, 35].”
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022, methodology): “We use the train sets of SNLI (Bow-man et al., 2015), ANLI, and FEVER-NLI, as well as the augmentation set generated via T AILOR (Ross et al., 2022), which perturbed SNLI hypotheses to create examples with high lexical overlap between the premise and hypothesis, and the augmentation set Z-Aug (Wu et”
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, methodology): “Generative models have also been used to aid experts with writing contrast sets (Wu et al., 2021; Ross et al., 2021), but to the best of our knowledge, this is the ﬁrst work to investigate the use of generative annotation assistants for crowdwork-ers directly in the annotation loop for NLP. Recent…”
Perturbation Augmentation for Fairer NLP (2022, background): “, 2021a), and some work even use learned methods for doing data augmentation for syntactic alternatives (Ross et al., 2022).”
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond (2021, background): “Indeed, in cases where counterfactual texts can be generated, we can often estimate causal effects on text-based models (Ribeiro et al., 2020; Gardner et al., 2020; Rosenberg et al., 2021; Ross et al., 2021; Meng et al., 2022; Zhang et al., 2022).”
How Many Data Samples is an Additional Instruction Worth? (2022, methodology): “They are allowed to use automated tools such as Semantic Control (Ross et al., 2021), Text Style Transfer (Reif et al.”