Detecting egregious responses in neural sequence-to-sequence models

By Tianxing He, James R. Glass. 2018

23 citations recorded by the source corpus.

In this work, we attempt to answer a critical question: whether there exists some input sequence that will cause a well-trained discrete-space neural network sequence-to-sequence (seq2seq) model to generate egregious outputs (aggressive, malicious, attacking, etc.). And if such inputs exist, how to find them efficiently. We adopt an empirical methodology, in which we first create lists of egregious output sequences, and then design a discrete optimization algorithm to find input sequences that will cause the model to generate them. Moreover, the optimization algorithm is enhanced for large vocabulary search and constrained to search for input sequences that are likely to be input by real-world users. In our experiments, we apply this approach to dialogue response generation models trained on three real-world dialogue data-sets: Ubuntu, Switchboard and OpenSubtitles, testing whether the model can generate malicious responses. We demonstrate that given the trigger inputs our algorithm finds, a significant number of malicious sentences are assigned large probability by the model, which reveals an undesirable consequence of standard seq2seq training.

Subfields: Biophysics, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Natural Language Processing

Citations in the corpus

Red Teaming Language Models with Language Models (2022, background): “, 2019, inter alia) and offensive text from LMs (Wallace et al., 2019; He and Glass, 2019; Liu et al., 2019; Song et al., 2020; Liu et al., 2020; Yu and Sagae, 2021).”
Universal Adversarial Triggers for Attacking and Analyzing NLP (2019, background): “He and Glass (2019) and Cheng et al. (2018) do the same for text generation.”
Chat as Expected: Learning to Manipulate Black-box Neural Dialogue Models (2020, background): “Besides, unlike some related works (He and Glass, 2018; Cheng et al., 2018) where crafted text inputs are ungrammatical and meaningless, the inputs generated by our model are smooth and natural utterances, which is guaranteed by the language model.”
Negative Training for Neural Dialogue Response Generation (2019, methodology): “samplemin-hit (Similar results are reported in (He and Glass, 2019)), while for Ubuntu and OpenSubti-”
Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots (2022, methodology): “Adversarial examples aim to craft queries that trigger a target chatbot providing exact or similar responses pre-defined by the adversary [21, 31].”
Adversarial Attacks and Defenses in Images, Graphs and Text: A Review (2019, background): “In recent years, many works (Szegedy et al., 2013; Goodfellow et al., 2014b; He et al., 2016) have shown that DNN models are vulnerable to adversarial examples, which can be formally defined as – “Adversarial examples are inputs to machine learning models that an attacker intentionally designed to…”
Hierarchical Reinforcement Learning for Open-Domain Dialog (2019, background): “Training on standard dialog datasets collected on-line or from movie scripts often leads to malicious, aggressive, biased, or offensive responses (Curry and Rieser 2018; He and Glass 2018; Henderson et al. 2018; Wallace et al. 2019).”
Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey (2019, methodology): “els. They used the Ubuntu Dialogue Corpus [115] and Dynamic Knowledge Graph Network with the Collaborative Communicating Agents (CoCoA) dataset [55] for the evaluation of their two attack strategies. [85] also used Ubuntu Dialogue Corpus. In addition, they also used Switchboard Dialogue Act Corpus21”
Say What I Want: Towards the Dark Side of Neural Dialogue Models (2019, methodology): “algorithm 2 Figure 1: The agent-environment setup of the proposed framework. indeed fails to lead the dialogue model to output the exact targeted responses. It should be pointed out that similar with [21], our work is not a model attack task in the true sense since we only focus on the requirements ”
Mix-review: Alleviate Forgetting in the Pretrain-Finetune Framework for Neural Language Generation Models (2019, unknown): “As shown by He & Glass (2019a), it is relatively difficult to trigger the dialogue models trained from scratch to output malicious responses (note that the conversations from the Dailydialogue data tend to be very polite).”
Automatically Exposing Problems with Neural Dialog Models (2021, background): “The most similar work to ours is probably directing pre-trained models into generating a list of pre-defined tokens or sentences (He and Glass, 2018; Liu et al., 2019a, 2020; He and Glass, 2020).”