Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?

By Sorami Hisamoto, Matt Post, Kevin Duh. 2019

132 citations recorded by the source corpus.

Data privacy is an important issue for “machine learning as a service” providers. We focus on the problem of membership inference attacks: Given a data sample and black-box access to a model’s API, determine whether the sample existed in the model’s training data. Our contribution is an investigation of this problem in the context of sequence-to-sequence models, which are important in applications such as machine translation and video captioning. We define the membership inference problem for sequence generation, provide an open dataset based on state-of-the-art machine translation models, and report initial results on whether these models leak private information against several kinds of membership inference attacks.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Extracting Training Data from Large Language Models (2020, background): “The least revealing form of attack is the membership inference attack [24, 42, 60, 62]: given a trained model, an adversary can predict whether or not a particular example was used to train the model.”
Red Teaming Language Models with Language Models (2022, methodology): “Data leakage can be used to infer the data used in training (“membership inference”; Shokri et al., 2017; Song and Shmatikov, 2019; Nasr et al., 2019; Hisamoto et al., 2020; Carlini et al., 2021), helping adversaries clone private, commercial LMs, violating intellectual property (Ateniese et al.,…”
A Study of Face Obfuscation in ImageNet (2021, background): “Membership inference attack infers whether an individual was included in training (Shokri et al., 2017; Nasr et al., 2019; Hisamoto et al., 2020).”
Membership Inference Attacks on Machine Learning: A Survey (2021, background): “[69] Embedding Wikipedia [128] 150,000 N.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “As the model is pretrained over a large amount of text data, it is possible to recover sensitive data including personally identifiable information [294]–[297].”
Thieves on Sesame Street! Model Extraction of BERT-based APIs (2019, methodology): “We use the logits in addition to the final layer representations of the victim model as input features to train the classifier, as model confidence scores and rare word representations are useful for membership inference (Song & Shmatikov, 2019; Hisamoto et al., 2019).”
A Survey of Privacy Attacks in Machine Learning (2020, methodology): “One such example is the ability to audit black-box models in order to see if data have been used without the data owner’s authorization [35, 86].”
Membership Inference Attacks against Language Models via Neighbourhood Comparison (2023, background): “Further studies of interest include work by Hisamoto et al. (2020), which studies membership inference attacks in machine translation, as well as work by Mireshghallah et al. (2022a), which investigates Likelihood Ratio Attacks for masked language models.”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, background): “Memorization in Language Models : Unintended memorization is a known challenge for language models [14, 85], which makes them open to extraction attacks [15, 89] and membership inference attacks [41, 64], although there has been work on mitigating these vulnerabilities [51, 88].”