Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls

By Da Ju, Jing Xu, Y-Lan Boureau, Jason Aaron Edward Weston. 2022

19 citations recorded by the source corpus.

The promise of interaction between intelligent conversational agents and humans is that models can learn from such feedback in order to improve. Unfortunately, such exchanges in the wild will not always involve human utterances that are benign or of high quality, and will include a mixture of engaged (helpers) and unengaged or even malicious users (trolls). In this work we study how to perform robust learning in such an environment. We introduce a benchmark evaluation, SafetyMix, which can evaluate methods that learn safe vs. toxic language in a variety of adversarial settings to test their robustness. We propose and analyze several mitigating learning algorithms that identify trolls either at the example or at the user level. Our main finding is that user-based methods, that take into account that troll users will exhibit adversarial behavior across multiple examples, work best in a variety of settings on our benchmark. We then test these methods in a further real-life setting of conversations collected during deployment, with similar results.

Subfields: Artificial Intelligence, Sociology and Political Science, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Safety, Robustness & Interpretability

Citations in the corpus

When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, methodology): “We also tested the zero-shot trans-ferability of both the satisfaction classiﬁers and the reply correctors on a real deployment dataset DEMO (Ju et al., 2022).”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “In a further companion paper (Ju et al., 2022) a study is conducted of how to robustly learn from dialogue data that may contain adversarial conversations and/or human feedback.”
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback (2022, background): “We make some steps to address this issue in a separate study (Ju et al., 2022).”