Learning Cross-Modal Embeddings for Cooking Recipes and Food Images

By Amaia Salvador, Nick Hynes, Yusuf Aytar, Javier Marín, Ferda Ofli, Ingmar Weber, Antonio Torralba. 2017

316 citations recorded by the source corpus.

In this paper, we introduce Recipe1M, a new large-scale, structured corpus of over 1m cooking recipes and 800k food images. As the largest publicly available collection of recipe data, Recipe1M affords the ability to train high-capacity models on aligned, multi-modal data. Using these data, we train a neural network to find a joint embedding of recipes and images that yields impressive results on an image-recipe retrieval task. Additionally, we demonstrate that regularization via the addition of a high-level classification objective both improves retrieval performance to rival that of humans and enables semantic vector arithmetic. We postulate that these embeddings will provide a basis for further exploration of the Recipe1M dataset and food and cooking in general. Code, data and models are publicly available

Subfields: Computer Vision and Pattern Recognition, Public Health, Environmental and Occupational Health, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, methodology): “Figure 4 presents the test O-F1 metric and cardinal- on COCO, VOC and Recipe1M. ity prediction errors for the baseline models FFBCE and FFsIoU as well as the models leading to the best performance for each dataset.”
Self-Supervised Moving Vehicle Tracking With Stereo Sound (2019, background): “Audio-visual data offers a wealth of resources for knowledge transfer between different modalities [5, 12, 34, 36].”
Zero-Shot Anticipation for Instructional Activities (2018, methodology): “Recipes usually include an ingredient list which is a rich source of information that can also serve as a strong modelling cue [25, 46].”
Cycle-Consistent Inverse GAN for Text-to-Image Synthesis (2021, methodology): “Recipe1M dataset is a public food dataset with large diversity, which has complex and fine-grained details on various classes of food images.”
RecipeNLG: A Cooking Recipes Dataset for Semi-Structured Text Generation (2020, methodology): “One of the results was the publication of the Recipe1M+ (Salvador et al., 2017) (Marin et al., 2019) dataset containing both recipes and images.”
Milvus: A Purpose-Built Vector Data Management System (2021, methodology): “per entity, we then use another dataset called Recipe1M [50, 56] that includes more than one million cooking recipes and food images.”
RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes (2018, methodology): “We employed the method proposed in (Salvador et al., 2017) to detect recipe ingredients.”
Recipe1M: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images (2018, methodology): “Since we presented our initial work on the topic back in 2017 [12], several related studies have been published and we feel obliged to provide a brief discussion about them.”
A Survey on Food Computing (2018, methodology): “[Salvador et al. 2017] developed a hybrid neural network architecture, which jointly learned shared space via image and recipe embedding for cross-modal image-recipe retrieval, where visual features are learned by CNN while recipe features are learned by LSTM. [Micael et al. 2018] extended…”
Self-Supervised Learning from Web Data for Multimodal Retrieval (2019, unknown): “In a more speciﬁc application, Salvador et al. [21] propose a joint embedding of food images and their recipes to identify ingredients, using Word2Vec [12] and LSTM representations to encode ingredient names and cooking instructions and a CNN to extract visual features from the associated images.”
WildFish: A Large Benchmark for Fish Recognition in the Wild (2018, methodology): “Strategy 69.0 RACNN [14] 65.4 OPAM [29] 71.6 Im2Recipe [31] 71.6 all the categories.”