Benchmarking Hierarchical Script Knowledge
By Yonatan Bisk, Jan Buys, Karl Pichotta, Yejin Choi. 2019
11 citations recorded by the source corpus.
Understanding procedural language requires reasoning about both hierarchical and temporal relations between events. For example, “boiling pasta” is a sub-event of “making a pasta dish”, typically happens before “draining pasta,” and requires the use of omitted tools (e.g. a strainer, sink...). While people are able to choose when and how to use abstract versus concrete instructions, the NLP community lacks corpora and tasks for evaluating if our models can do the same. In this paper, we introduce KidsCook, a parallel script corpus, as well as a cloze task which matches video captions with missing procedural details. Experimental results show that state-of-the-art models struggle at this task, which requires inducing functional commonsense knowledge not explicitly stated in text.
Subfields: Artificial Intelligence, Language and Linguistics, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yejin Choi
- Mohit Bansal
- Jianfeng Gao
- Graham Neubig
- Luke S. Zettlemoyer
- Noah A. Smith
- Ronan Le Bras
- Hannaneh Hajishirzi
In Language and Linguistics
- Xuhui Zhou
- Yejin Choi
- Graham Neubig
- Dan Jurafsky
- Jianfeng Gao
- Rowan Zellers
- Yonatan Bisk
- Jillian Grennan
In Natural Language Processing
- Yejin Choi
- Jianfeng Gao
- Mohit Bansal
- Ronan Le Bras
- Graham Neubig
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Rowan Zellers
In Evaluation & Benchmarks
- Yejin Choi
- Ronan Le Bras
- Jianfeng Gao
- Rowan Zellers
- Yonatan Bisk
- Mohit Bansal
- Dan Jurafsky
- Luke S. Zettlemoyer
Citations in the corpus
- PIQA: Reasoning about Physical Commonsense in Natural Language (2019, background): “…about cause and effect effects/implications within these models (Bosse-lut et al. 2019), extracting knowledge from them (Petroni et al. 2019), and investigating where large scale language models fail to capture knowledge of tools and elided procedural knowledge in recipes (Bisk et al. 2019).”
- WebArena: A Realistic Web Environment for Building Autonomous Agents (2023, background): “Each task is described as a high-level natural language intent, emulating the abstract language usage patterns typically employed by humans [2].”
- Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts (2019, background): “, 2016, 2017), or as a prompt for users to generate descriptions (Bisk et al., 2019).”