Inverse Cooking: Recipe Generation From Food Images

By Amaia Salvador, Michal Drozdzal, Xavier Giró-i-Nieto, Adriana Romero. 2018

170 citations recorded by the source corpus.

People enjoy food photography because they appreciate food. Behind each meal there is a story described in a complex recipe and, unfortunately, by simply looking at a food image we do not have access to its preparation process. Therefore, in this paper we introduce an inverse cooking system that recreates cooking recipes given food images. Our system predicts ingredients as sets by means of a novel architecture, modeling their dependencies without imposing any order, and then generates cooking instructions by attending to both image and its inferred ingredients simultaneously. We extensively evaluate the whole system on the large-scale Recipe1M dataset and show that (1) we improve performance w.r.t. previous baselines for ingredient prediction; (2) we are able to obtain high quality recipes by leveraging both image and ingredients; (3) our system is able to produce more compelling recipes than retrieval-based approaches according to human judgment. We make code and models publicly available.

Subfields: Computer Vision and Pattern Recognition, Food Science, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Food Science

In Multimodal Learning

Citations in the corpus

Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, methodology): “This has been done, for example, by using recurrent neural networks, at the expense of introducing intrinsic label ordering during training, which has been resolved by applying a category-wise max-pooling across the time dimension [62, 7, 71, 54] or by optimizing for the most likely ground truth lab”
RecipeNLG: A Cooking Recipes Dataset for Semi-Structured Text Generation (2020, background): “In particular, there is a number of interesting endeavors utilizing computer vision for ﬁnding (Salvador et al., 2017) or even generating (Salvador et al., 2019) cooking recipes matching the input food image.”
On the Evaluation of Conditional GANs (2019, background): “Among the most promising approaches, Variational Auto-Encoders (VAEs) [20], auto-regressive models [43, 42] and Generative Adversarial Networks (GANs) [9] have been driving significant progress, with the latter at the forefront of a wide-range of applications [24, 52, 32, 46, 1, 38, 34].”