COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis

By Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, Jie Zhou. 2019

448 citations recorded by the source corpus.

There are substantial instruction videos on the Internet, which enables us to acquire knowledge for completing various tasks. However, most existing datasets for instruction video analysis have the limitations in diversity and scale, which makes them far from many real-world applications where more diverse activities occur. Moreover, it still remains a great challenge to organize and harness such data. To address these problems, we introduce a large-scale dataset called “COIN" for COmprehensive INstruction video analysis. Organized with a hierarchical structure, the COIN dataset contains 11,827 videos of 180 tasks in 12 domains (e.g., vehicles, gadgets, etc.) related to our daily life. With a new developed toolbox, all the videos are annotated effectively with a series of step descriptions and the corresponding temporal boundaries. Furthermore, we propose a simple yet effective method to capture the dependencies among different steps, which can be easily plugged into conventional proposal-based action detection methods for localizing important steps in instruction videos. In order to provide a benchmark for instruction video analysis, we evaluate plenty of approaches on the COIN dataset under different evaluation criteria. We expect the introduction of the COIN dataset will promote the future in-depth research on instruction video analysis for the community.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, positive): “Next, we evaluate our visual representation on COIN [73] action segmentation task in Table 4a.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “In a similar vein, COIN [50] and CrossTask [68] datasets are collected by first searching for tasks on WikiHow and then videos for each task on YouTube.”
ActBERT: Learning Global-Local Video-Text Representations (2020, methodology): “The action segmentation task in COIN is to design an action label for a video at the frame-level.”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, neutral): “The recent inﬂux of instructional video datasets such as How2 [28], Inria Instructional Videos [60], COIN [61], CrossTask [31], YouCook2 [23], Mining YouTube [62], and HowTo100M [25] has inspired a variety of methods for semi-supervised text-video modelling.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, methodology): “Frame Accuracy ↑ S UPERVISED NN-Viterbi (Richard et al., 2018) 21.2 VGG (Simonyan and Zisserman, 2014) 25.8 TCFPN-ISBA (Ding and Xu, 2018) 34.3 CBT (Sun et al., 2019a) 53.9 ActBERT (Zhu and Yang, 2020) 57.0 MIL-NCE (Miech et al., 2020) 61.0 VideoCLIP (Fine-tuned) 68.7 Z ERO - SHOT VideoCLIP (Zero-sh”
Look Before you Speak: Visually Contextualized Utterances (2020, methodology): “We set benchmarks on both the HowTo100M and COIN datasets, and show state-of-the-art results on downstream video QA benchmarks.”
Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities (2022, background): “Multi-step activities are best exemplified in cooking and instructional videos, so the majority of datasets in this area are curated from online video platforms, e.g. YouTube Instructional [1], What’s Cooking [27], YoucookII [50], CrossTask [52], COIN [49] and HowTo100M [29].”
Speech2Action: Cross-Modal Supervision for Action Recognition (2020, background): “Recently, a number of works have creatively identified certain domains of videos, such as narrated instructional videos [27, 38, 51] and lifestyle vlogs [11, 17] that are available in huge numbers (e.”
Condensed Movies: Story Based Retrieval with Contextual Embeddings (2020, background): “A number of different works have recently creatively identified that certain domains of videos, such as narrated instructional videos [6,7,8] and lifestyle vlogs [9,10] are available ar X iv :2 00 5.”
Human Action Recognition and Prediction: A Survey (2018, background): “COIN (Tang et al., 2019) 2019 11,827 1 180 – RGB Uncontrolled”
VideoMamba: State Space Model for Efficient Video Understanding (2024, unknown): “In conclusion, our experiments reveal VideoMamba’s potential in understanding both short-term (K400 [37] and SthSthV2 [24]) and long-term (Breakfast [38], COIN [72], and LVU [86]) video contents.”
Learning Video Representations using Contrastive Bidirectional Transformer (2019, background): “(Right) Action segmentation results on the COIN dataset (Tang et al., 2019).”
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities (2020, background): “Additionally, prior instructional video datasets have either drastic view perspective changes [65,1,55,57] or limited egocentric view with severe occlusions [41,36], hindering the activity understanding.”
Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube (2020, background): “A number of multimodal instructional video datasets have been proposed (Wang et al., 2019; Tang et al., 2019; Sanabria et al., 2018).”
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment (2021, methodology): “We apply TACo and different variants of contrastive losses to train or pretrain and ﬁne-tune on various downstream tasks including text-video retrieval (YouCook2, MSR-VTT and ActivityNet) [58, 52, 12], video action step localization (CrossTask) [61] and action segmentation (COIN) [43].”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “COIN30 (Tang et al., 2019) is a large-scale dataset of instructional YouTube videos from 12 domains such as vehicles, gadgets, sports, etc.”
TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (2023, background): “It also incorporates 12 specific datasets [6, 8, 9, 11, 14, 15, 31, 41, 43, 55, 56, 60] derived from different domains as illustrated in Fig.”