Information Leakage in Embedding Models

By Congzheng Song, Ananth Raghunathan. 2020

376 citations recorded by the source corpus.

Embeddings are functions that map raw input data to low-dimensional vector representations, while preserving important semantic information about the inputs. Pre-training embeddings on a large amount of unlabeled data and fine-tuning them for downstream tasks is now a de facto standard in achieving state of the art learning in many domains. We demonstrate that embeddings, in addition to encoding generic semantics, often also present a vector that leaks sensitive information about the input data. We develop three classes of attacks to systematically study information that might be leaked by embeddings. First, embedding vectors can be inverted to partially recover some of the input data. As an example, we show that our attacks on popular sentence embeddings recover between 50%--70% of the input words (F1 scores of 0.5--0.7). Second, embeddings may reveal sensitive attributes inherent in inputs and independent of the underlying semantic task at hand. Attributes such as authorship of text can be easily extracted by training an inference model on just a handful of labeled embedding vectors. Third, embedding models leak moderate amount of membership information for infrequent training data inputs. We extensively evaluate our attacks on various state-of-the-art embedding models in the text domain. We also propose and evaluate defenses that can prevent the leakage to some extent at a minor cost in utility.

Subfields: Artificial Intelligence, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Privacy & Security

Citations in the corpus

Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, background): “Later model inversion attacks focus on fully recreating training samples given access to only a trained model (Hidano et al., 2017; Song & Raghunathan, 2020; Yang et al., 2019).”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “To mitigate this risk, researchers have developed strategies like differential privacy, curated training data (Car-lini et al., 2019, 2021), and introducing auxiliary loss functions (Song and Raghunathan, 2020; Li et al., 2022a).”
Extracting Training Data from Large Language Models (2020, background): “Separately, model inversion attacks [21] [8,66,68,73], or settings where the adversary has a priori knowledge of the secret they want to extract (e.g., a social security number) [8,27].”
What Does it Mean for a Language Model to Preserve Privacy? (2022, background): “The power of membership inference attacks have been demonstrated on natural language processing models such as NLP classiﬁers [83] as well as released embeddings [87, 61].”
Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots (2022, background): “NLP systems also face safety issues in various scenarios, including information leakage [57], offensive content generation [20], etc.”
Enhanced Membership Inference Attacks against Machine Learning Models (2021, background): “, 2020], sentence embeddings [Song and Raghunathan, 2020], and speech recognition models [Shah et al.”
A Survey of Privacy Attacks in Machine Learning (2020, background): “Such an active attacker reached an attack accuracy of 92.1%.”
Large Language Model Alignment: A Survey (2023, background): “Attack Methods against Alignment Privacy Attacks Jailbreaking Prompts Li et al. (2023a)Deng et al. (2023) Backdoor Attacks Prompt Injection Liu et al. (2023e) Zhao et al. (2023a) Greshake et al. (2023) Kandpal et al. (2023) Backdoors Injection at RLHF Shi et al. (2023) Adversarial Attacks Adversaria”
Membership Inference Attacks on Machine Learning: A Survey (2021, background): “Song and Raghunathan [182] introduced the first MIAs on word embedding and sentence embedding models.”
Multi-step Jailbreaking Privacy Attacks on ChatGPT (2023, background): “On the other hand, a few works (Li et al., 2022; Pan et al., 2020; Song and Raghunathan, 2020) examined information leakage issues on LMs’ embeddings during inference time.”
Security and Privacy Challenges of Large Language Models: A Survey (2024, neutral): “Publication date: February 2024. a baseline approach that completely relies on the loss of the target model, as established in previous work ([115], [126]).”
Membership Inference Attacks against Language Models via Neighbourhood Comparison (2023, background): “…user data introduces a variety of privacy risks: While embedding models have been shown to leak personal attributes of their data (Song and Raghunathan, 2020), generative language models are capable of generating verbatim repetitions of their training data and therefore exposing sensitive…”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “Attribute inference Attack [199, 172, 128, 249, 174, 153] is a type of threat where an attacker attempts to deduce sensitive or personal information of individuals or entities by analyzing the behavior or responses of a machine learning models.”