Engaging Image Captioning via Personality

By Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bordes, Jason Aaron Edward Weston. 2018

162 citations recorded by the source corpus.

Standard image captioning tasks such as COCO and Flickr30k are factual, neutral in tone and (to a human) state the obvious (e.g., “a man playing a guitar”). While such tasks are useful to verify that a machine understands the content of an image, they are not engaging to humans as captions. With this in mind we define a new task, PERSONALITY-CAPTIONS, where the goal is to be as engaging to humans as possible by incorporating controllable style and personality traits. We collect and release a large dataset of 241,858 of such captions conditioned over 215 possible traits. We build models that combine existing work from (i) sentence representations [36] with Transformers trained on 1.7 billion dialogue examples; and (ii) image representations [32] with ResNets trained on 3.5 billion social media images. We obtain state-of-the-art performance on Flickr30k and COCO, and strong performance on our new task. Finally, online evaluations validate that our task and models are engaging to humans, with our best model close to human performance.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “This is a continuation of the approach taken in multiple papers using role-played situations (Rashkin et al., 2019), assigned traits (Shuster et al., 2018, 2019), assigned movie preferences (Kang et al., 2019), or even an entire fantasy universe (Urbanek et al., 2019; Prabhumoye et al., 2020).”
Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, background): “…dialogue agent’s models are built on the Transformer architecture (Vaswani et al., 2017), which has been shown to perform well on a variety of NLP tasks (Devlin et al., 2018; Radford et al., 2018), including multiple persona-based chat applications (Shuster et al., 2018a,b; Rashkin et al., 2018).”
Controlling Style in Generated Dialogue (2020, methodology): “Shuster et al. (2019) proposes a dataset of images captioned with a target style, while Shuster et al. (2018) provides short conversations with target styles.”
Captioning Images Taken by People Who Are Blind (2020, background): “Over the past decade, nearly 20 publicly-shared captioning datasets have been created to support the development of automated captioning algorithms [6, 14, 15, 17, 18, 20, 24, 25, 26, 29, 30, 40, 43, 50, 51, 54].”
CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021, background): “5B) vs. TransRes-Net (ResNet-152) (see Shuster et al. (2019) Table 7, Row 5), but no better than a length-only base-line (also 57%).”
Engaging Image Chat: Modeling Personality in Grounded Dialogue (2018, methodology): “C HAT and a generalization of the model introduced in (Shuster et al., 2018) to the case of dialogue.”
From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “Some datasets have been collected to explore this variant, such as InstaPIC [236], which is composed of multiple Instagram posts from the same users, FlickrStyle10K [239], which contains images and textual sentences with two different styles, and Personality-Captions [243],”
NICE: Neural Image Commenting with Empathy (2021, methodology): “pendix E show an experiment that we adapt trained MAGIC model to Personality-Captions dataset (PCD) (Shuster et al., 2019).”
Complementary Auxiliary Classifiers for Label-Conditional Text Generation (2020, background): “• Personality captioning (Shuster et al. 2019) was proposed for engaging image captioning via personality.”
All-in-One Image-Grounded Conversational Agents (2019, methodology): “…of skills, a large set of imageand-text tasks have been devised by researchers, covering image captioning [Young et al., 2014; Chen et al., 2015; Shuster et al., 2019a], visual question answering [Goyal et al., 2017; Das et al., 2017], and dialogue based on an image [Mostafazadeh et al., 2017;…”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Furthermore, to make the generated captions more human like, personality traits (Shuster et al., 2018) where explored to generate captions.”
Image-Chat: Engaging Grounded Conversations (2018, methodology): “Style Traits A number of works have shown that style traits for image captioning help provide creative captions (Mathews et al., 2018, 2016; Gan et al., 2017; Shuster et al., 2019).”
From Show to Tell: A Survey on Image Captioning (2021, background): “…multiple Instagram posts from the same users, FlickrStyle10K [210], which contains images and textual sentences with two different styles, and Personality-Captions [214], which contains triples of images, captions, and one among 215 personality traits to be used to condition the caption generation.”
Language Models Can See: Plugging Visual Controls in Text Generation (2022, background): “Meanwhile, some controllable image captioning methods [48, 21, 8, 70, 7] propose to generate diverse descriptions by feeding different control signals (e.g., label and text), which require additional annotations for training.”