Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12

By Jun Tang, Aleksandra Korolova, Xiaolong Bai, Xueqiang Wang, Xiaofeng Wang. 2017

321 citations recorded by the source corpus.

In June 2016, Apple announced that it will deploy differential privacy for some user data collection in order to ensure privacy of user data, even from Apple. The details of Apple's approach remained sparse. Although several patents have since appeared hinting at the algorithms that may be used to achieve differential privacy, they did not include a precise explanation of the approach taken to privacy parameter choice. Such choice and the overall approach to privacy budget use and management are key questions for understanding the privacy protections provided by any deployment of differential privacy. In this work, through a combination of experiments, static and dynamic code analysis of macOS Sierra (Version 10.12) implementation, we shed light on the choices Apple made for privacy budget management. We discover and describe Apple's set-up for differentially private data processing, including the overall data pipeline, the parameters used for differentially private perturbation of each piece of data, and the frequency with which such data is sent to Apple's servers. We find that although Apple's deployment ensures that the (differential) privacy loss per each datum submitted to its servers is $1$ or $2$, the overall privacy loss permitted by the system is significantly higher, as high as $16$ per day for the four initially announced applications of Emojis, New words, Deeplinks and Lookup Hints. Furthermore, Apple renews the privacy budget available every day, which leads to a possible privacy loss of 16 times the number of days since user opt-in to differentially private data collection for those four applications. We advocate that in order to claim the full benefits of differentially private data collection, Apple must give full transparency of its implementation, enable user choice in areas related to privacy loss, and set meaningful defaults on the privacy loss permitted.

Subfields: Artificial Intelligence, Sociology and Political Science, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Privacy & Security

Citations in the corpus

Honeycrisp: large-scale differentially private aggregation without a trusted core (2019, background): “However, no finite budget would be enough to support periodic queries indefinitely, which is why Apple opted to replenish the budget every day [72].”
Differentially-Private "Draw and Discard" Machine Learning (2018, methodology): “Since differential privacy started to gain traction in industry, there has been significant debate devoted to establishing what is a reasonable privacy loss rate and to optimizing the analyses and algorithms to reduce the privacy loss needed [25], [49], [43].”
Collecting Telemetry Data Privately (2017, background): “In the context of telemetry collection one typically considers algorithms that exhibit differential privacy in the local model [12, 14, 7, 5, 3, 18], also called randomized response model [19], γ-amplification [13], or FRAPP [1].”
Prochlo: Strong Privacy for Analytics in the Crowd (2017, background): “Local differential privacy is the approach taken by our RAPPOR software monitoring system for the Chrome Web browser [28].”
BLENDER: Enabling Local Search with a Hybrid Differential Privacy Model (2017, background): “, by what is a common practice for differentially private algorithms in the industry [36, 5, 18]).”
Encode, Shuffle, Analyze Privacy Revisited: Formalizations and Empirical Evaluation (2020, background): “Finally, the option of simply replicating the details of prominent LDP-reporting deployments is not very attractive, since these have been criticized both for a lack of privacy and a lack of utility [1], [2], [20].”
When Relaxations Go Bad: "Differentially-Private" Machine Learning (2019, unknown): “Successful practical implementations are now available for private querying on databases [49] and collecting user statistics from web browsers [19,67].”
Context-Aware Generative Adversarial Privacy (2017, background): “Indeed, it was recently reported that Apple's DP implementation su ers from several limitations most notable of which is Apple's use of unacceptably large leakage parameters [79].”
Edge Computing Security: State of the Art and Challenges (2019, unknown): “Existing research showed that Apple’s implementation of differential privacy results in a high privacy loss of 16 per day, while the ideal loss should not be greater than 2 [92].”
Privacy-Preserving Blockchain-Based Federated Learning for IoT Devices (2019, methodology): “For example, Apple’s mobile operating system iOS utilizes DP algorithms in [11].”
Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference (2019, methodology): “For this reason, in practice, ε is often chosen to be quite large; for example, in 2017, Apple was found to use an effective epsilon as high as 16 in some of its routines [43].”