Chat as Expected: Learning to Manipulate Black-box Neural Dialogue Models

By Haochen Liu, Tyler Derr, Jiliang Tang. 2020

15 citations recorded by the source corpus.

Recently, neural network based dialogue systems have become ubiquitous in our increasingly digitalized society. However, due to their inherent opaqueness, some recently raised concerns about using neural models are starting to be taken seriously. In fact, intentional or unintentional behaviors could lead to a dialogue system to generate inappropriate responses. Thus, in this paper, we investigate whether we can learn to craft input sentences that result in a black-box neural dialogue model being manipulated into having its outputs contain target words or match target sentences. We propose a reinforcement learning based model that can generate such desired inputs automatically. Extensive experiments on a popular well-trained state-of-the-art neural dialogue model show that our method can successfully seek out desired inputs that lead to the target outputs in a considerable portion of cases. Consequently, our work reveals the potential of neural dialogue models to be manipulated, which inspires and opens the door towards developing strategies to defend them.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “Liu et al. (2020) demonstrate this by generating prompts that manipulated an E2E model to generate outputs containing predeﬁned offensive terms.”
Recipes for Safety in Open-domain Chatbots (2020, background): “In the dialogue domain, Liu et al. (2020) show how an RL-based approach can hone in on prompts that would lead an unprotected model to output a number of responses deemed undesirable.”
Red Teaming Language Models with Language Models (2022, background): “, 2019, inter alia) and offensive text from LMs (Wallace et al., 2019; He and Glass, 2019; Liu et al., 2019; Song et al., 2020; Liu et al., 2020; Yu and Sagae, 2021).”
Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation (2022, background): “Liu et al. (2019, 2020); Yu and Sagae (2021) used reinforcement learning to find contexts that could elicit toxic responses from the dialogue models.”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, methodology): “Liu et al. (2020) demonstrate this by generating prompts that manipulated an E2E model to generate outputs containing offensive terms.”
Mitigating Gender Bias for Neural Dialogue Generation with Adversarial Learning (2020, background): “A biased dialogue system will produce improper speeches, which can bring in bad experiences to users or even cause negative social impacts (Wolf et al., 2017; Liu et al., 2019b, 2020).”
Automatically Exposing Problems with Neural Dialog Models (2021, background): “The most similar work to ours is probably directing pre-trained models into generating a list of pre-deﬁned tokens or sentences (He and Glass, 2018; Liu et al., 2019a, 2020; He and Glass, 2020).”