Zero-Shot Anticipation for Instructional Activities

By Fadime Sener, Angela Yao. 2018

75 citations recorded by the source corpus.

How can we teach a robot to predict what will happen next for an activity it has never seen before? We address the problem of zero-shot anticipation by presenting a hierarchical model that generalizes instructional knowledge from large-scale text-corpora and transfers the knowledge to the visual domain. Given a portion of an instructional video, our model predicts coherent and plausible actions multiple steps into the future, all in rich natural language. To demonstrate the anticipation capabilities of our model, we introduce the Tasty Videos dataset, a collection of 2511 recipes for zero-shot learning, recognition and anticipation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Li Fei-Fei
3. Chelsea Finn
4. Abhinav Kumar Gupta
5. Trevor Darrell
6. Pieter Abbeel
7. Kristen Grauman
8. Antonio Torralba

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Kristen Grauman
3. Li Fei-Fei
4. Jitendra Malik
5. Trevor Darrell
6. Cordelia Schmid
7. Andrew Zisserman
8. Dacheng Tao

Citations in the corpus

• Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, methodology): “, CLIP [71]) have been trained on joint image(s) and corresponding text captions via variants of a masked language modeling objective [72, 73, 74, 75], a contrastive loss [76, 77, 71] or other supervised objectives[78, 79].”
• Procedure Planning in Instructional Videos (2019, background): “g tasks. Some of the tasks ask the question \What is happening?", such as action recognition and temporal action segmentation [4, 13, 28], state understanding [2], video summarization/captioning [29,32,37], retrieval [32] etc. The others ask the question \What is going to happen?", such as”
• Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities (2022, background): “• Goal-oriented free-style procedures: Existing datasets feature multi-step activities following a strictly ordered recipe [28, 40, 50, 52] or script [8, 12, 34, 43, 47].”
• PlaTe: Visually-Grounded Planning With Transformers in Procedural Tasks (2021, background): “Action label prediction [25], [26] addresses the problem of anticipating all activities within a time horizon.”
• Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, methodology): “In several papers, recurrent neural networks are used in a straightforward way: a single RNN serves as a video aggregation module [63, 103, 58].”
• Action Modifiers: Learning From Adverbs in Instructional Videos (2019, background): “More recently, instructional videos are a popular source of datasets [1, 33, 44, 60] with hundreds of online videos of the same task.”