Investigating the Impact of Pre-trained Word Embeddings on Memorization in Neural Networks

39 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Quantifying Memorization Across Neural Language Models (2022, background): “, 2020; Ziegler, 2021), or in other work, synthetically injected “canaries” (Carlini et al., 2019; Henderson et al., 2017; Thakkar et al., 2020; Thomas et al., 2020).”
Counterfactual Memorization in Neural Language Models (2021, methodology): “…work on memorization in neural language models demonstrated the ability to extract memorized training data, including sensitive data such as phone numbers and usernames [Carlini et al., 2020, Ziegler, 2021, Carlini et al., 2019, Henderson et al., 2017, Thakkar et al., 2020, Thomas et al., 2020].”
Bag of Tricks for Training Data Extraction from Language Models (2023, background): “Despite being a relatively new task, many of the underlying technologies and analysis methods, including membership inference (Shokri et al., 2017) and leveraging network memorization for attacks (Thomas et al., 2020; Leino & Fredrikson, 2020), were introduced much earlier.”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, background): “Memorization in Language Models: Unintended memorization is a known challenge for language models [12, 13], which makes them open to extraction attacks [14, 15] and membership inference attacks [16, 17], although there has been work on mitigating these vulnerabilities [11, 18].”