What’s Cookin’? Interpreting Cooking Videos using Text, Speech and Vision

By Jonathan Malmaud, Jonathan Huang, Vivek Rathod, Nick Johnston, Andrew Rabinovich, Kevin P. Murphy. 2015

163 citations recorded by the source corpus.

We present a novel method for aligning a sequence of instructions to a video of someone carrying out a task. In particular, we focus on the cooking domain, where the instructions correspond to the recipe. Our technique relies on an HMM to align the recipe steps to the (automatically generated) speech transcript. We then refine this alignment using a state-of-the-art visual food detector, based on a deep convolutional neural network. We show that our technique outperforms simpler techniques based on keyword spotting. It also enables interesting applications, such as automatically illustrating recipes with keyframes, and searching within a video for events of interest.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

VideoBERT: A Joint Model for Video and Language Representation Learning (2019, background): “, [16, 2, 10, 38, 39]) have trained models to analyse instructional videos, such as cooking.”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, positive): “To avoid labeling visual data, several approaches have leveraged audio transcripts obtained from narrated videos using automatic speech recognition (ASR) as a way to supervise video models for object detection [4, 16, 56], captioning [34, 72], classification [3, 44, 49, 89], summarization [59] or re”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “In particular, instructional videos [2, 30, 68] often contain narration with an explicit intention of explaining the visual content on screen.”
Joint Discovery of Object States and Manipulation Actions (2017, background): “in instructional videos [3, 32, 37] or while performing daily activities [38].”
Unsupervised Learning from Narrated Instruction Videos (2015, background): “While this work analyses cooking videos at a great scale, it relies on readily-available recipes which may not be available for more general scenarios.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Prior works have used web videos+ASR to build weakly-supervised object detectors [75], action detectors/classifiers [108, 5, 51, 72], instruction aligners [65, 4, 15], video captioners [85, 38, 74, 89], and visual reference resolvers [41].”
Towards Automatic Learning of Procedures From Web Instructional Videos (2017, methodology): “It emphasizes video-subtitle alignment (Malmaud et al. 2015; Bojanowski et al. 2015) and discovery of common procedure steps of a specific process (Alayrac et al.”
Grounding of Textual Phrases in Images by Reconstruction (2015, background): “ks on spatial-temporal language grounding are [22] and [39]. These are again limited to small set of nouns. There have been a number of works aiming to temporally align language descriptions to video [2, 24, 30, 41]. However, these works do not perform any spatial grounding for language descriptions”
Unsupervised Semantic Parsing of Video Collections (2015, methodology): “In addition to the language based approaches, Malmaud et al.[38] consider both language and vision modalities and propose a method to align an input video to a recipe.”
Connectionist Temporal Modeling for Weakly Supervised Action Labeling (2016, background): “Our work is also related to recent progress on using instructional videos or movie scripts [2,4,9,25,30,35,51] as supervision for video parsing.”
A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions (2019, background): “formative in a variety of instructional video understanding tasks (Naim et al., 2014, 2015; Malmaud et al., 2015; Sener et al., 2015; Alayrac et al., 2016; Huang et al., 2017); though less work has focused on instructional caption generation, which”
TutorialVQA: Question Answering Dataset for Tutorial Videos (2019, background): “…the narratives with the visual information may Overall Given the video in top 10 0.1579 0.6385 Table 5: Segment level prediction for Baseline3 pipeline, both overall and given that the video is in the top 10. improve the performance, similarly to the text alignment task in (Malmaud et al., 2015).”
Towards a Dataset for Human Computer Communication via Grounded Language Acquisition (2016, background): “…between less formulaic language and simple actions has been explored successfully in the context of simulated worlds (Branavan et al. 2009; Goldwasser and Roth 2011; Branavan, Silver, and Barzilay 2011; Artzi and Zettlemoyer 2013) and videos (Malmaud et al. 2015; Venugopalan et al. 2015).”
Grounding Object Detections With Transcriptions (2019, methodology): “The labelled video frames can be used to fine-tune a VGG-16 object classification convolutional neural network model. et al., 2014; Malmaud et al., 2015).”