Understanding and Predicting Image Memorability at a Large Scale

By Aditya Khosla, Akhil S. Raju, Antonio Torralba, Aude Oliva. IEEE International Conference on Computer Vision, 2015

352 citations recorded by the source corpus.

Progress in estimating visual memorability has been limited by the small scale and lack of variety of benchmark data. Here, we introduce a novel experimental procedure to objectively measure human memory, allowing us to build LaMem, the largest annotated image memorability dataset to date (containing 60,000 images from diverse sources). Using Convolutional Neural Networks (CNNs), we show that fine-tuned deep features outperform all other features by a large margin, reaching a rank correlation of 0.64, near human consistency (0.68). Analysis of the responses of the high-level CNN layers shows which objects and regions are positively, and negatively, correlated with memorability, allowing us to create memorability maps for each image and provide a concrete method to perform image memorability manipulation. This work demonstrates that one can now robustly estimate the memorability of images from many different classes, positioning memorability and deep memorability features as prime candidates to estimate the utility of information for cognitive systems. Our model and data are available at: http://memorability.csail.mit.edu.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0e89a45cb30f3e488fa14f74cc8ee7a209199638

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers (2021, methodology): “Metrics raw attention LIME [42] MemNet [29] GradCAM [44] LRP [4] Ours pixel accuracy 67.”
Domain2Vec: Domain Embedding for Unsupervised Domain Adaptation (2020, methodology): “Color [37] 10,000 pedestrian 24 LabelMeFacade [38] 395 building 25 UT Zappos50K [39] 10,000 shoes 26 TRANCOS [40] 1,244 traffic 27 FGVC [40] 10,000 aeroplane 28 Mall Dataset [41] 2,000 mall 29 Chars74K [42] 7,705 character 30 DomainNet [8] 10,000 painting 31 Paris Dataset [43] 3,187 street 32 Domain”
StyleGAN2 Distillation for Feed-forward Image Manipulation (2020, methodology): “age manipulations in a variety of domains and with various GAN architectures. In GANalyze [16], the attention is directed to search interpretable directions in latent space of BigGAN [6] using MemNet [31] as \assessor" network. Jahanian et al. [26] show that walk in latent space lead to interpr”
Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts (2019, background): “…of a post (Soleymani et al., 2017), or its perlocutionary force (how it is perceived by its audience), including aspects such as memorability (Khosla et al., 2015), saliency (Bylinskii et al., 2018), popularity (Khosla et al., 2014) and virality (Deza and Parikh, 2015; Alameda-Pineda et al.,…”
The Role of Typicality in Object Classification: Improving The Generalization Capacity of Convolutional Neural Networks (2016, background): “The last row of the fourth box, indicates that ﬁne-tuning AlexNet with the memorability score [ Khosla et al. , 2015 ] will increase its generalization performance (comparing to baselines).”
Eye Tracking for Everyone (2016, methodology): “The recent success of deep learning has been apparent in a variety of domains in computer vision [20, 7, 36, 28, 19], but its impact on improving the performance of eye tracking has been rather limited [43].”
GANalyze: Toward Visual Definitions of Cognitive Image Properties (2019, methodology): “Finally, α acts as knob to set the degree of change one wants to achieve in the Assessor value (e.g., MemNet score).”
Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text (2018, methodology): “s for ads is to be remembered. Therefore a reasonable guess is that visual memorability can affect how designers arrange elements within ads, including both image and text. Here we adopt the model in [19] and perform inference on our collected ads, and the 1D resultant memorability score (higher mea”