Robust Text Classification in the Presence of Confounding Bias

By Virgile Landeiro, Aron Culotta. 2016

50 citations recorded by the source corpus.

As text classifiers become increasingly used in real-time applications, it is critical to consider not only their accuracy but also their robustness to changes in the data distribution. In this paper, we consider the case where there is a confounding variable Z that influences both the text features X and the class variable Y. For example, a classifier trained to predict the health status of a user based on their online communications may be confounded by socioeconomic variables. When the influence of Z changes from training to testing data, we find that classifier accuracy can degrade rapidly. Our approach, based on Pearl's back-door adjustment, estimates the underlying effect of a text variable on the class variable while controlling for the confounding variable. Although our goal is prediction, not causal inference, we find that such adjustments are essential to building text classifiers that are robust to confounding variables. On three diverse text classifications tasks, we find that covariate adjustment results in higher accuracy than competing baselines over a range of confounding relationships (e.g., in one setting, accuracy improves from 60% to 81%).

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, methodology): “These efforts include employing techniques from the causal inference literature that can lead to more robust research results (Landeiro and Culotta, 2016; Proserpio et al., 2016), or calibrating non-representative social data samples (Zagheni andWeber, 2015).”
A Survey of Learning Causality with Data (2018, background): “[73] and a causal feature selection method [98]. While for prediction under concept drift [141], where Eq.47is violated but the marginal distribution of predictors remain, causal knowledge also helps [76]. In addition, for causal discovery, one can detect causes for the target variable through searc”
Counterfactual Fairness in Text Classification through Robustness (2018, background): “ variety of techniques, such as adversarial training [4, 16] and adversarial logit pairing [8]. Several papers draw connections between fairness, text generation, and robustness. Landeiro and Culotta [13] consider robustness in text with respect to counfounding variables such as the author’s gender,”
Distilling the Outcomes of Personal Experiences: A Propensity-scored Analysis of Social Media (2017, background): “Relatedly, Landeiro and Culotta apply causal modeling techniques to build more robust classifiers for social media texts [58].”
Fair Forests: Regularized Tree Induction to Minimize Model Bias (2017, result): “This makes our results comparable with a larger body of work, as their approach and metrics have been widely used through the literature [1, 6, 10, 19].”
A Survey of Learning Causality with Data: Problems and Methods (2018, background): “47 is violated but the marginal distribution of predictors remain, causal knowledge also helps [76].”