Generate Your Counterfactuals: Towards Controlled Counterfactual Generation for Text

By Nishtha Madaan, Inkit Padhi, Naveen Panwar, Diptikalyan Saha. 2020

117 citations recorded by the source corpus.

Machine Learning has seen tremendous growth recently, which has led to a larger adaptation of ML systems for educational assessments, credit risk, healthcare, employment, criminal justice, to name a few. The trustworthiness of ML and NLP systems is a crucial aspect and requires a guarantee that the decisions they make are fair and robust. Aligned with this, we propose a novel framework GYC, to generate a set of exhaustive counterfactual text, which are crucial for testing these ML systems. Our main contributions include a) We introduce GYC, a framework to generate counterfactual samples such that the generation is plausible, diverse, goal-oriented, and effective, b) We generate counterfactual samples, that can direct the generation towards a corresponding \texttt{condition} such as named-entity tag, semantic role label, or sentiment. Our experimental results on various domains show that GYC generates counterfactual text samples exhibiting the above four properties. GYC generates counterfactuals that can act as test cases to evaluate a model and any text debiasing algorithm.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Tailor: Generating and Perturbing Text with Semantic Controls (2021, methodology): “, training a style transferer requires instances labeled with positive and negative sentiments (Madaan et al., 2020b).”
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, methodology): “Similar to Madaan et al. (2021), we compare the diversity and closeness of Polyjuice with alternative generators, i.e., RoBERTa and T5, representing masked language models that prioritize word and span substitution, and original GPT-2, representing the standard generative model not conditioned on x.”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, background): “…research direction is the use of causal in-272 ference for measuring biases in LLMs, for example to analyze if the generated text by an LLM is affected considerably by only changing the protected attributes or categories in the input (Huang et al., 2020; Madaan et al., 2021; Cheng et al., 2021).”
Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review (2020, unknown): “Chen et al. [63] and De Toni et al. [80] use RL to generate CFE as was also proposed by Verma et al. [337].”
Explaining NLP Models via Minimal Contrastive Editing (MiCE) (2020, background): “Madaan et al. (2021) applies con-such as free-text rationales (Liang et al., 2020) or highlights (Jacovi and Goldberg, 2020).”