You Don’t Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers’ Private Personas

By Haoran Li, Yangqiu Song, Lixin Fan. 2022

23 citations recorded by the source corpus.

Social chatbots, also known as chit-chat chatbots, evolve rapidly with large pretrained language models. Despite the huge progress, privacy concerns have arisen recently: training data of large language models can be extracted via model inversion attacks. On the other hand, the datasets used for training chatbots contain many private conversations between two individuals. In this work, we further investigate the privacy leakage of the hidden states of chatbots trained by language modeling which has not been well studied yet. We show that speakers’ personas can be inferred through a simple neural network with high accuracy. To this end, we propose effective defense objectives to protect persona leakage from hidden states. We conduct extensive experiments to demonstrate that our proposed defense objectives can greatly reduce the attack accuracy from 37.6% to 0.5%. Meanwhile, the proposed objectives preserve language models’ powerful generation ability.

Subfields: Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

1. Zhexin Zhang
2. Hao Sun
3. Jie Huang
4. Minlie Huang
5. Yangqiu Song
6. Jiawen Deng
7. Jiale Cheng
8. Haoran Li

In Safety, Robustness & Interpretability

1. Zhexin Zhang
2. Hao Sun
3. Jie Huang
4. Minlie Huang
5. Jiawen Deng
6. Jiale Cheng

Citations in the corpus

• Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “To mitigate this risk, researchers have developed strategies like differential privacy, curated training data (Car-lini et al., 2019, 2021), and introducing auxiliary loss functions (Song and Raghunathan, 2020; Li et al., 2022a).”
• The rise and potential of large language model based agents: a survey (2025, unknown): “Such situations could lead to unauthorized surveillance, data breaches, and misuse of personal information, especially when subjected to malicious attacks [473].”
• Multi-step Jailbreaking Privacy Attacks on ChatGPT (2023, background): “On the other hand, a few works (Li et al., 2022; Pan et al., 2020; Song and Raghunathan, 2020) examined information leakage issues on LMs’ embeddings during inference time.”