Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images

By Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, Bernt Schiele. 2017

423 citations recorded by the source corpus.

With an increasing number of users sharing information online, privacy implications entailing such actions are a major concern. For explicit content, such as user profile or GPS data, devices (e.g. mobile phones) as well as web services ( e.g. facebook) offer to set privacy settings in order to enforce the users’ privacy preferences. We propose the first approach that extends this concept to image content in the spirit of a Visual Privacy Advisor. First, we categorize personal information in images into 68 image attributes and collect a dataset, which allows us to train models that predict such information directly from images. Second, we run a user study to understand the privacy preferences of different users w.r.t. such attributes. Third, we propose models that predict user specific privacy score from images in order to enforce the users’ privacy preferences. Our model is trained to predict the user specific privacy risk and even outperforms the judgment of the users, who often fail to follow their own privacy preferences on image data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Sociology and Political Science, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Sociology and Political Science

In Privacy & Security

Citations in the corpus

VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, neutral): “Reinforcement learning enables model optimization with non-differentiable evaluation metrics [14, 45, 61, 64].”
A Distributional Approach to Controlled Text Generation (2020, methodology): “Continuous approximation using the Gumbel Softmax was developed for the training of Variational Autoencoders but several works have implemented it for natural language generation Shetty et al. (2017); Chu & Liu (2019); Kusner & Hernández-Lobato (2016).”
Object Hallucination in Image Captioning (2018, positive): “We additionally consider a model trained with a GAN loss (Shetty et al., 2017) (denoted GAN ), which applies adversarial training to obtain more diverse and “human-like” captions, and their respective non-GAN baseline with the CE loss.”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “al network and deep recurrent network based model. Dai et al. [26] also proposed a GAN based image captioning method. However, they do not consider multiple captions for a single image. Shetty et al. [126] introduced a new GAN based image captioning method. This method can generate multiple captions”
Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioning (2019, background): “Nonetheless, in recent years, remarkable progress has been made in image captioning [3,6,7,12–14,21,22,24, 25, 30, 36, 39, 41] and particularly controllable [10, 40] and diverse [9,10,26,34,38,40] image captioning.”
Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions (2018, neutral): “Some works have extended the beam-search algorithm to sample multiple captions from the same distribution [41, 1], while different GAN-based approaches have also appeared [8, 39, 45].”
Knockoff Nets: Stealing Functionality of Black-Box Models (2018, background): “ries? 4.What makes for a good knockoff architecture? 2. Related Work Privacy, Security and Computer Vision. Privacy has been largely addressed within the computer vision community by proposing models [28,30,31,42,49,50] which recognize and control privacy-sensitive information in visual content. The”
Language Generation with Recurrent Generative Adversarial Networks without Pre-training (2017, background): “Consequently, past work on using GANs for text generation has been based on pre-training (Yu et al., 2016; Li et al., 2017; Yang et al., 2017; Wu et al., 2017; Liang et al., 2017; Zhang et al., 2016; Shetty et al., 2017)”
Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model (2017, neutral): “image captioning [6, 41], dialog generation [23], or text generation [53] – by either viewing the generative model as a stochastic parametrized policy that is updated using REINFORCE”
Towards Privacy-Preserving Visual Recognition via Adversarial Training: A Pilot Study (2018, methodology): “Instead of using all 68 attributes in [39], we find that many of them rarely appear in UCF-101 (shown in the supplementary).”
Understanding and Controlling User Linkability in Decentralized Learning (2018, methodology): “, [19, 40, 54, 73] for AP and [16, 29, 38] for Top-1/5 accuracy.”
Sequence to Sequence Mixture Model for Diverse Machine Translation (2018, neutral): “Some work puts their efforts on decoding stages, and form a group of beam search to encourage diversity (Vijayakumar et al., 2016), while others pay more attention to adversarial training (Shetty et al., 2017; Li et al., 2018).”
Gradient-Leaks: Understanding and Controlling Deanonymization in Federated Learning (2018, background): “While such large-scale data advances learning fundamental ML models (e.g., visual object recognition), the representations also encode a massive amount of unnecessary individual-speciﬁc information (e.g., person identities) [33,61].”
Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention (2017, neutral): “• Percentage of dierent sentences: percentage of images which are described dierently by the two models; en, we measure the diversity of the set of captions generated by each of the two models, via the following two metrics [40]: • Div-1: ratio of number of unique unigrams in a set of captions to”
pix2code: Generating Code from a Graphical User Interface Screenshot (2017, neutral): “Generative Adversarial Networks GANs [7] have shown to be extremely powerful at generating images and sequences [23, 15, 25, 17, 3].”
When Machine Learning Meets Privacy (2020, methodology): “d ML: (a) Privacy of ML model and data; (b) ML enhanced privacy protection; (c) ML-based privacy attack. al. [82] utilized ML to enhance private decision-making experience through ML. Orekondy et al. [114] proposed an approach to categorize personal information in images and predict information leak”
Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text (2018, neutral): “…in making literal connections between image and text (e.g., image captioning, where the text describes what is seen in the image [1, 6, 9, 16, 18, 35, 38, 40]), recognizing implicit relationships between image and text (e.g., metaphorical, symbolic, explanatory, ironic, etc.) remains a research…”
Say As You Wish: Fine-Grained Control of Image Caption Generation With Abstract Scene Graphs (2020, neutral): “What is worse, such passive caption generation can greatly hinder diversity and tend to generate mediocre descriptions [34, 38].”