Say What I Want: Towards the Dark Side of Neural Dialogue Models

By Haochen Liu, Tyler Derr, Zitao Liu, Jiliang Tang. 2019

17 citations recorded by the source corpus.

Neural dialogue models have been widely adopted in various chatbot applications because of their good performance in simulating and generalizing human conversations. However, there exists a dark side of these models -- due to the vulnerability of neural networks, a neural dialogue model can be manipulated by users to say what they want, which brings in concerns about the security of practical chatbot services. In this work, we investigate whether we can craft inputs that lead a well-trained black-box neural dialogue model to generate targeted outputs. We formulate this as a reinforcement learning (RL) problem and train a Reverse Dialogue Generator which efficiently finds such inputs for targeted outputs. Experiments conducted on a representative neural dialogue model show that our proposed model is able to discover such desired inputs in a considerable portion of cases. Overall, our work reveals this weakness of neural dialogue models and may prompt further researches of developing corresponding solutions to avoid it.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiliang Tang
2. Minlie Huang
3. Haochen Liu
4. Tyler Derr
5. Michael Backes
6. Zitao Liu
7. Jeremy Blackburn
8. Hui Liu

In Computer Vision and Pattern Recognition

1. Jiliang Tang
2. Haochen Liu
3. Tyler Derr
4. Michael Backes
5. Hao Sun
6. Nat McAleese
7. Dian Yu
8. Emiliano De Cristofaro

In Natural Language Processing

1. Jiliang Tang
2. Haochen Liu
3. Minlie Huang
4. Hui Liu
5. Tyler Derr
6. Zitao Liu
7. Zhexin Zhang
8. Jamell Dacon

In Safety, Robustness & Interpretability

1. Jiliang Tang
2. Haochen Liu
3. Hui Liu
4. Zitao Liu
5. Zhexin Zhang
6. Tyler Derr
7. Jamell Dacon
8. Michael Backes

Citations in the corpus

• Does Gender Matter? Towards Fairness in Dialogue Systems (2019, background): “Offensive responses cause users discomfort and should be avoided (Henderson et al., 2018; Dinan et al., 2019b; Liu et al., 2019; Liu et al., 2020b).”
• Red Teaming Language Models with Language Models (2022, background): “, 2019, inter alia) and offensive text from LMs (Wallace et al., 2019; He and Glass, 2019; Liu et al., 2019; Song et al., 2020; Liu et al., 2020; Yu and Sagae, 2021).”
• Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation (2022, background): “Liu et al. (2019, 2020); Yu and Sagae (2021) used reinforcement learning to find contexts that could elicit toxic responses from the dialogue models.”
• Chat as Expected: Learning to Manipulate Black-box Neural Dialogue Models (2020, background): “(10) with the intermediate reward to accelerate the training speed and effectiveness (Liu et al., 2017).”
• Adversarial Attacks and Defenses in Images, Graphs and Text: A Review (2019, background): “The work (Liu et al., 2019) treats the neural dialogue model as a black-box and adopts a reinforcement learning framework to effectively find trigger inputs for targeted responses.”
• Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots (2022, methodology): “A similar idea is to craft adversarial examples against chatbot models using the reinforcement learning approach [31].”
• Automatically Exposing Problems with Neural Dialog Models (2021, background): “The most similar work to ours is probably directing pre-trained models into generating a list of pre-defined tokens or sentences (He and Glass, 2018; Liu et al., 2019a, 2020; He and Glass, 2020).”