Visual Storytelling

By Ting-Hao 'Kenneth' Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross B. Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra and 5 more. 2016

543 citations recorded by the source corpus.

We introduce the first dataset for sequential vision-to-language, and explore how this data may be used for the task of visual storytelling. The first release of this dataset, SIND1 v.1, includes 81,743 unique photos in 20,211 sequences, aligned to both descriptive (caption) and story language. We establish several strong baselines for the storytelling task, and motivate an automatic metric to benchmark progress. Modelling concrete description as well as figurative and social language, as provided in this dataset and the storytelling task, has the potential to move artificial intelligence from basic understandings of typical visual scenes towards more and more human-like understanding of grounded event structure and subjective expression.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “A.2 Details of the language model Our language model follows standard best practices: the input and output embedding layers are tied (Inan et al., 2017; Press and Wolf, 2017), all embedding and hidden layers are set to 512, and we used recurrent dropout (Gal and Ghahramani, 2016) on the hidden…”
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, background): “[56] for a survey), with great progress on important problems such as visual question answering [3, 26] and image caption generation and retrieval [8, 88, 43, 68, 28], with offshoot tasks focusing specifically on visual reasoning [36], referring expressions [39], visual storytelling [57, 32], visual”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “In this section, we show more examples of MERLOT unshufﬂing visual stories in SIND [50, 33].”
Visual Dialog (2016, background): “A number of problems at the intersection of vision and language have recently gained prominence – image captioning [12, 13, 23, 56], video/movie description [45, 53, 54], text-to-image coreference/grounding [8, 19, 24, 39, 41, 44], visual storytelling [2, 20], and of course, visual question answerin”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “Recently, a new dataset was proposed providing both concrete conceptual and ‘narrative’ texts coupled with images (Huang et al., 2016), a promising new direction for this branch of nlg.”
Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models (2016, background): “improvements in posterior-probabilities not necessarily corresponding to improvements in task-specific metrics), it is common practice to deliberately throttle BS to become a poorer optimization algorithm by using reduced beam widths (Vinyals et al., 2015; Karpathy & Fei-Fei, 2015; Ferraro et al., 2”
Plan-And-Write: Towards Better Automatic Storytelling (2018, background): “ structures (Mostafazadeh et al. 2016b; McDowell et al. 2017). Rishes et al. [2013] presents a model that reproduce different versions of a story from its symbolic representation. Pichotta and Mooney [2016] parse a large collection of natural language documents, extract sequences of events, and lear”
Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation (2018, background): “vision and language research has attracted a lot of attention. Much work [38,31,15,9,40,34,33,32] has been done in language generation conditioned on visual inputs. There is also another line of work [14,4] that tries to answer questions from images. The task of vision-language grounding [30,2,1] is”
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners (2022, background): “Our inspiration is derived from the fact that humans are excellent visual storytellers [15], with the ability to piece together a coherent story from a few isolated images.”
Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation (2017, background): “…have enabled much interdisciplinary research in vision and language, from video transcription (Rohrbach et al., 2012; Venugopalan et al., 2015), to answering questions about images (Antol et al., 2015; Malinowski and Fritz, 2014), to storytelling around series of photographs (Huang et al., 2016).”
Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation (2019, background): “ence descriptions, generating long and semantic-coherent stories or reports to describe visual contents (e.g., images, videos) has recently attracted increasing research interests (Liang et al. 2017; Huang et al. 2016; Krause et al. 2017), and is posed as a more challenging and realistic goal toward”
Generating Images with Multimodal Language Models (2023, methodology): “In addition to our evaluations on VIST [28] and VisDial [16], we also run evaluations on our model’s ability to generate images from MS-COCO [34] captions (Tab.”
Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos (2017, background): “on of reference resolution as graph optimization is related to the long-standing effort of extracting graphs from image/video. This includes recent progress in scene graphs [13,20,51, 64], storylines [1,14,17,31,53], and action understanding [10,44,55]. Our approach of extracting graph associating t”
Text-to-image Diffusion Models in Generative AI: A Survey (2023, background): “Moreover, ARLDM [111] shows the consistency for unseen characters, and also the ability for real-world story synthesis on a new introduced dataset VIST [112].”