Deepdiary: Lifelogging image captioning and summarization

By Chenyou Fan, Zehua Zhang, David J. Crandall. 2018

157 citations recorded by the source corpus.

Abstract Automatic image captioning has been studied extensively over the last few years, driven by breakthroughs in deep learning-based image-to-text translation models. However, most of this work has considered captioning web images from standard data sets like MS-COCO, and has considered single images in isolation. To what extent can automatic captioning models learn finer-grained contextual information specific to a given person’s day-to-day visual experiences? In this paper, we consider captioning image sequences collected from wearable, life-logging cameras. Automatically-generated captions could help people find and recall photos among their large-scale life-logging photo collections, or even to produce textual “diaries” that summarize their day. But unlike web images, photos from wearable cameras are often blurry and poorly composed, without an obvious single subject. Their content also tends to be highly dependent on the context and characteristics of the particular camera wearer. To address these challenges, we introduce a technique to jointly caption sequences of photos, which allows captions to take advantage of temporal constraints and evidence across time, and we introduce a technique to increase the diversity of generated captions, so that they can describe a photo from multiple perspectives (e.g., first-person versus third-person). To test these techniques, we collect a dataset of about 8000 realistic lifelogging images, a subset of which are annotated with nearly 5000 human-generated reference sentences. We evaluate the quality of image captions both quantitatively and qualitatively using Amazon Mechanical Turk, finding that while these algorithms are not perfect, they could be an important step towards helping to organize and summarize lifelogging photos.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jiayi Ma
3. Andy Zeng
4. Junchi Yan
5. Kristen Grauman
6. Dacheng Tao
7. Li Fei-Fei
8. Jitendra Malik

In Multimodal Learning

1. Michael S. Ryoo
2. Chenyou Fan
3. Junchi Yan
4. Andy Zeng

Citations in the corpus

• Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “Relatively less research has been carried out specifically on egocentric image captioning [118].”
• Image Matching from Handcrafted to Deep Features: A Survey (2020, background): “Meanwhile, image stitching or image mosaic involves obtaining a wider field-of-view of a scene from a sequence of partial views (Ghosh and Kaabouch 2016).”
• EgoVQA - An Egocentric Video Question Answering Benchmark Dataset (2019, background): “More and more people are using wearable cameras for various applications such as lifelogging [13,22,30], children behavior study [8,9], sports video analysis [11,12], event detection [50], etc.”