TGIF: A New Dataset and Benchmark on Animated GIF Description

By Yuncheng Li, Yale Song, Liangliang Cao, Joel R. Tetreault, Larry Goldberg, Alejandro Jaimes, Jiebo Luo. 2016

314 citations recorded by the source corpus.

With the recent popularity of animated GIFs on social media, there is need for ways to index them with rich meta-data. To advance research on animated GIF understanding, we collected a new dataset, Tumblr GIF (TGIF), with 100K animated GIFs from Tumblr and 120K natural language descriptions obtained via crowdsourcing. The motivation for this work is to develop a testbed for image sequence description systems, where the task is to generate natural language descriptions for animated GIFs or video clips. To ensure a high quality dataset, we developed a series of novel quality controls to validate free-form text input from crowd-workers. We show that there is unambiguous association between visual content and natural language descriptions in our dataset, making it an ideal benchmark for the visual content captioning task. We perform extensive statistical analyses to compare our dataset to existing image and video description datasets. Next, we provide baseline results on the animated GIF description task, using three representative techniques: nearest neighbor, statistical machine translation, and recurrent neural networks. Finally, we show that models fine-tuned from our animated GIF description dataset can be helpful for automatic movie description.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, neutral): “Charades [48] 10k 16k 10,000 82h Home 2016 MSR-VTT [58] 10k 200k 7,180 40h Youtube 2016 YouCook2 [67] 14k 14k 2,000 176h Youtube 2018 EPIC-KITCHENS [7] 40k 40k 432 55h Home 2018 DiDeMo [15] 27k 41k 10,464 87h Flickr 2017 M-VAD [52] 49k 56k 92 84h Movies 2015 MPII-MD [43] 69k 68k 94 41h Movies 2015 A”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “Unlike UCF-101, VideoGPT did not overﬁt on TGIF with a train loss of 2 .”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “STVQA [18, 17] advances EVQA by applying two duallayer LSTMs for video and question modelling, with additional spatio-temporal attention modules for visual reasoning.”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, background): “with subtitles; MovieQA [75], with videos from movies and questions obtained from higher-level plot summaries; PororoQA [43], with cartoon videos; and TGIFQA [39], with templated questions from the TGIF dataset [47].”
Movie Description (2016, methodology): “TGIF is a large dataset of 100k image sequences (GIFs) with associated descriptions.”
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering (2017, methodology): “The TGIF dataset utilizes animated GIFs as their visual data, which have recently emerged as an attractive source of data in computer vision [13, 22] due to their concise format and cohesive storytelling nature [5]; this makes it especially ideal for vision and language understanding.”
Localizing Moments in Video with Natural Language (2017, neutral): “TGIF [19] 100,000 /125,781 Tumblr GIFs Open”
Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering (2019, methodology): “TGIF-QA [10] contains 165K QA pairs associated with 72K GIF images based on the TGIF dataset [14].”
Dual Encoding for Zero-Example Video Retrieval (2018, methodology): “As TRECVID does not specify training data for the AVS task, we train the dual encoding network using the joint collection of MSR-VTT and the TGIF [17], which contains 100K animated GIFs and 120K sentences describing visual content of the GIFs.”
Video Question Answering with Spatio-Temporal Reasoning (2019, methodology): “The automatic validation technique by Li et al. (2016) is originally proposed for collecting free-form sentences for video captioning.”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “We evaluate on three popular public benchmarks: MSR-VTT [92], MSVD [97], and TGIF [98].”
PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning (2024, methodology): “First, the open-ended Video Question Answer (VideoQA) includes MSVD-QA [39], MSRVTT-QA [39], ActivityQA [43], and TGIF QA [20].”
Recent Advances in Video Question Answering: A Review of Datasets and Methods (2021, methodology): “TGIF-QA dataset [44] contains∼ 165K QA pairs from ∼ 72K animated GIFs taken from the TGIF dataset.”
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, background): “The video training data contains a subset of VideoChat2-IT (Li et al., 2024b), which includes TextVR (Wu et al., 2025), Youcook2 (Zhou et al., 2018), Kinetics-710 (Kay et al., 2017), NExTQA (Xiao et al., 2021), CLEVRER (Yi et al., 2019), EgoQA (Fan, 2019), TGIF (Li et al., 2016), WebVidQA (Yang et a”
Video Question Answering via Hierarchical Spatio-Temporal Attention Networks (2017, methodology): “We construct the video question-answering datset from the annotated video clip data [Li et al., 2016] with natural language descriptions, which consists of 201,068 GIFs and 287,933 descriptions.”