RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes

By Semih Yagcioglu, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis. 2018

196 citations recorded by the source corpus.

Understanding and reasoning about cooking recipes is a fruitful research direction towards enabling machines to interpret procedural text. In this work, we introduce RecipeQA, a dataset for multimodal comprehension of cooking recipes. It comprises of approximately 20K instructional recipes with multiple modalities such as titles, descriptions and aligned set of images. With over 36K automatically generated question-answer pairs, we design a set of comprehension and reasoning tasks that require joint understanding of images and text, capturing the temporal flow of events and making sense of procedural knowledge. Our preliminary results indicate that RecipeQA will serve as a challenging test bed and an ideal benchmark for evaluating machine comprehension systems. The data and leaderboard are available at http://hucvl.github.io/recipeqa.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Experience Grounds Language (2020, background): “, 2012), a myriad of tasks on visual question answering (Antol et al., 2015; Das et al., 2018; Yagcioglu et al., 2018), natural language and visual reasoning (Suhr et al.”
Datasheets for datasets (2018, neutral): “Academic researchers have adopted our proposal and released datasets with accompanying datasheets [e.g., 7, 10, 23, 26].”
Visuo-Lingustic Question Answering (VLQA) Challenge (2020, background): “Further, we obtained a subset of interesting samples from existing datasets Figure 4: VLQA data creation process: collect data using primary and secondary sources, then perform post-processing (if any), then ﬁnally create question-answers that require joint reasoning. such as RecipeQA (Yagcioglu et ”
A Survey on Food Computing (2018, background): “Recently, [Semih et al. 2018] released a recipe dataset RecipeQA, which includes additional 36K questions to support question answering compared with other recipe datasets.”
Effect of Vision-and-Language Extensions on Natural Language Understanding in Vision-and-Language Models (2021, methodology): “…these models have recently been used for more challenging applications such as multi-modal machine reading comprehension (Kembhavi et al., 2017; Yagcioglu et al., 2018; Hannan et al., 2020; Tanaka et al., 2021) and multi-modal dialogue systems (Ilinykh et al., 2019; Haber et al., 2019; Udagawa…”
RecipeNLG: A Cooking Recipes Dataset for Semi-Structured Text Generation (2020, background): “(Yagcioglu et al., 2018) published a dataset consisting of approximately 20K recipes to generate question-answer pairs.”
MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering (2022, background): “…et al., 2019; Kwiatkowski et al., 2019; Yang et al., 2015) reading comprehension datasets (Yang et al., 2018; Lai et al., 2017; Zellers et al., 2018; Yagcioglu et al., 2018; Dua et al., 2019; Bajaj et al., 2018; Huang et al., 2019), extractive question answering (Hermann et al., 2015; Trischler et…”
Auditing ImageNet: Towards a Model-driven Framework for Annotating Demographic Attributes of Large-Scale Image Datasets (2019, background): “Since the initial release of this paper, several datasets have adopted this practice [27, 24], however work remains for wide adoption.”