Sort Story: Sorting Jumbled Images and Captions into Stories

By Harsh Agrawal, Arjun Chandrasekaran, Dhruv Batra, Devi Parikh, Mohit Bansal. 2016

62 citations recorded by the source corpus.

Temporal common sense has applications in AI tasks such as QA, multi-document summarization, and human-AI communication. We propose the task of sequencing -- given a jumbled set of aligned image-caption pairs that belong to a story, the task is to sort them such that the output sequence forms a coherent story. We present multiple approaches, via unary (position) and pairwise (order) predictions, and their ensemble-based combinations, achieving strong results on this task. We use both text-based and image-based features, which depict complementary improvements. Using qualitative examples, we demonstrate that our models have learnt interesting aspects of temporal common sense.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Table 2: Results unscrambling SIND visual stories[42, 1].”
Visual Dialog (2016, background): “As analyzed in [2,16,62], this leads to a particular bias in the questions – people only ask ‘Is there a clocktower in the picture?’ on pictures actually containing clock towers.”
RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes (2018, background): “Here, we should note that a similar task has been previously investigated by Agrawal et al. (2016) for visual stories where the task is to order a jumbled set of aligned image-description pairs.”
Making History Matter: Gold-Critic Sequence Training for Visual Dialog (2019, background): “So far, one may identify that the key difference between Visual Dialog (VisDial) and another well-known task Visual Question Answering (VQA) [3, 2, 6, 12, 1, 4, 6, 22] is the exploitation of the history .”
Making History Matter: History-Advantage Sequence Training for Visual Dialog (2019, background): “image caption [9, 10, 18, 2] and visual question answering [2, 1, 3, 5, 25]).”
It Takes Two to Tango: Towards Theory of AI's Mind (2017, background): “To the best of our knowledge, this is the first evaluation that measures the extent to which these modalities allow a human to build a mental model for the AI and predict its behavior.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “The task is also observed from a different perspective where given a jumbled set of aligned image-description pairs that belong to a story, the task is to sort them such that the output sequence forms a coherent story (Agrawal et al., 2016).”