Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack

By Emily Dinan, Samuel Humeau, Bharath Chintagunta, Jason Aaron Edward Weston. 2019

288 citations recorded by the source corpus.

The detection of offensive language in the context of a dialogue has become an increasingly important application of natural language processing. The detection of trolls in public forums (Galán-García et al., 2016), and the deployment of chatbots in the public domain (Wolf et al., 2017) are two examples that show the necessity of guarding against adversarially offensive behavior on the part of humans. In this work, we develop a training scheme for a model to become robust to such human attacks by an iterative build it, break it, fix it scheme with humans and models in the loop. In detailed experiments we show this approach is considerably more robust than previous systems. Further, we show that offensive language used within a conversation critically depends on the dialogue context, and cannot be viewed as a single sentence offensive detection task as in most previous work. Our newly collected tasks and methods are all made open source and publicly available.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Training language models to follow instructions with human feedback (2022, background): “There are many ways to mitigate these harms, including by fine-tuning on a small, value-targeted dataset (Solaiman and Dennison, 2021), filtering the pretraining dataset (Ngo et al., 2021), or human-in-the-loop data collection (Dinan et al., 2019; Xu et al., 2020).”
Experience Grounds Language (2020, background): “More flexible, dynamic evaluation (Zellers et al., 2020; Dinan et al., 2019) are a partial solution, but true persistence of identity and adaption to change are both necessary and still a long way off.”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “The “Build it Break it Fix it” strategy in the security domain (Ruef et al., 2016) has been adapted to NLP (Ettinger et al., 2017) as well as dialogue safety (Dinan et al., 2019).”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “This is partially due to detectors’ over-reliance on lexical cues of toxicity (including swearwords, slurs, and other “bad” words Dinan et al., 2019).”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “Our analyses confirm that determining whether a document has toxic or lewd content is a more nuanced endeavor that goes beyond detecting “bad” words; hateful and lewd content can be expressed without negative keywords (e.g., microaggressions, innuendos; Breitfeller et al., 2019; Dinan et al., 2019).”
HateCheck: Functional Tests for Hate Speech Detection Models (2020, background): “Social context and conversational context in particular have been shown to improve classification performance on hate speech detection tasks (Qian et al., 2018a; Dinan et al., 2019).”
OPT: Open Pre-trained Transformer Language Models (2022, background): “There has been a great deal of work on mitigations for toxicity and biases (Dathathri et al., 2019; Dinan et al., 2019a; Sheng et al., 2019; Dinan et al., 2020a; Liu et al., 2019a; Krause et al., 2020; Xu et al., 2020; Liang et al., 2021; Dinan et al., 2021; Xu et al., 2021a; Dhamala et al., 2021; S”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “Dinan et al. (2019) and Xu et al. (2020) both investigated the human-in-the-loop method to make adversarial attacks on the dialogue models.”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, methodology): “One potentially useful tool for addressing harm is red teaming—using manual or automated methods to adversarially probe a language model for harmful outputs, and then updating the model to avoid such outputs [42, 20, 3, 11].”
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, background): “Examples are collected at scale us-ing keyword-based scraping approaches (Waseem, 2016; Davidson et al., 2017; Zampieri et al., 2019), the bootstrapped scraping approaches (Founta et al., 2018), and machine-vs-human adversarial data collection (Dinan et al., 2019; Vidgen et al., 2021), among others.”
Understanding Dataset Difficulty with V-Usable Information (2021, positive): “Our findings corroborate prior work which shows that certain lexical items (e.g., swear words, identity mentions) are responsible for hate speech prediction (Dixon et al., 2018; Dinan et al., 2019).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “This might be because the safety classiﬁer was trained to identify dialog utterances that are “ not OK to send in a friendly conversation with someone you just met online ”, which may encapsulate more than just toxic responses (Dinan et al., 2019b).”
Recipes for Safety in Open-domain Chatbots (2020, methodology): “…collection setup are given in Appendix A. Figure 1 demonstrates how this adversarial set-up differs from the “Build-it, Break-it, Fix-it” set-up from Dinan et al. (2019b): namely, in the former, the “breaker” (or adversarial user) tries to break a classiﬁer by submitting human-authored…”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “Furthermore, we have consistently found that domain-specific pretraining is important to high performance: that is, using dialoguelike pretraining significantly outperforms generic pretraining on resources like Wikipedia and BooksCorpus (Dinan et al., 2019b; Humeau et al., 2019; Dinan et al., 2019a,”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “While several recent efforts have been made to describe and mitigate unsafe behaviour of conversational models (e.g. Dinan et al., 2019; Xu et al., 2021; Ouyang et al., 2022; Thoppilan et al., 2022; Perez et al., 2022; Dinan et al., 2022), this work aims to provide a framework to help practitioners…”
Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, methodology): “The classiﬁer was ﬁne-tuned on an o ﬀ ensive language classiﬁcation task (Dinan et al., 2019), and achieves state-of-the-art results.”
Does Gender Matter? Towards Fairness in Dialogue Systems (2019, methodology): “Offensive responses cause users discomfort and should be avoided (Henderson et al., 2018; Dinan et al., 2019b; Liu et al., 2019; Liu et al., 2020b).”
Improving alignment of dialogue agents via targeted human judgements (2022, background): “Though prior work has demonstrated that language and dialogue models can output harmful language (Brown et al., 2020; Dinan et al., 2019; Rae et al., 2021), our baseline produced language we consider harmful only infrequently or under adversarial behaviour by users.”
Red Teaming Language Models with Language Models (2022, methodology): “Prior work relies on human annotators to generate test cases (Dinan et al., 2019; Nie et al., 2020; Ribeiro et al., 2020; Röttger et al., 2021; Xu et al., 2021b; Wallace et al., 2021) and/or detect failures (Dinan et al., 2019; Ziegler et al., 2019; Nie et al., 2020; Stiennon et al., 2020; Xu et…”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “, 2017), Build-It Break-It Fix-It (BBF) (Dinan et al., 2019a) and Bot Adversarial Dialogue dataset (BAD) (Xu et al.”