Cooking with Semantics
By Jonathan Malmaud, Earl J. Wagner, Nancy Chang, Kevin P. Murphy. 2014
68 citations recorded by the source corpus.
We are interested in the automatic interpretation of how-to instructions, such as cooking recipes, into semantic representations that can facilitate sophisticated question answering. Recent work has shown impressive results on semantic parsing of instructions with minimal supervision, but such techniques cannot handle much of the situated and ambiguous language used in instructions found on the web. In this paper, we suggest how to extend such methods using a model of pragmatics, based on a rich representation of world state.
Subfields: Artificial Intelligence, Natural Language Processing, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Li Fei-Fei
- Trevor Darrell
- Mohit Bansal
- Ali Farhadi
- Abhinav Kumar Gupta
- Luc Van Gool
- Silvio Savarese
In Natural Language Processing
- Mohit Bansal
- Yonatan Bisk
- Dieter Fox
- Mohit Shridhar
- Roozbeh Mottaghi
- Luke S. Zettlemoyer
- Dipendra Kumar Misra
- Josef Sivic
In Multimodal Learning
- Trevor Darrell
- Lisa Anne Hendricks
- Juan Carlos Niebles
- Josef Sivic
- Mohit Shridhar
- De-An Huang
- Roozbeh Mottaghi
- Dieter Fox
Citations in the corpus
- ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “There, research has focused on mapping instructions to actions [5, 13, 31, 35, 48], but these works do not involve visual, interactive environments.”
- Localizing Moments in Video with Natural Language (2017, background): “lizing Video Segments with Natural Language. Prior work has considered aligning natural language with video, e.g., instructional videos with transcribed text (Kiddon et al., 2015; Huang et al., 2017; Malmaud et al., 2014, 2015). Our work is most related to recent work in video moment retrieval with ”
- Unsupervised Semantic Parsing of Video Collections (2015, methodology): “Recent methods on natural language processing [40, 58] focus on semantic parsing of language recipes in order to extract actions and the objects in the form of predicates.”
- RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes (2018, background): “Recent examples include parsing of recipes (Malmaud et al., 2014; Jermsurawong and Habash, 2015), aligning instructional text to videos (Mal-”
- Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos (2018, unknown): “During training, we leverage the Stanford CoreNLP parser [30] to automatically parse entities.”
- Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos (2017, background): “Our goal of resolving reference in transcription of instructional videos is related to the procedure text understanding in the NLP community [4, 18, 23, 29, 33, 34, 36].”
- Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions (2014, background): “Malmaud et al. (2014) noticed many ambiguous and incomplete instructions in the context of interpreting cooking recipes.”