Procedure Planning in Instructional Videos

By Chien-Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, Juan Carlos Niebles. 2019

129 citations recorded by the source corpus.

We propose a new challenging task: procedure planning in instructional videos. Unlike existing planning problems, where both the state and the action spaces are well-defined, the key challenge of planning in instructional videos is that both the state and the action spaces are open-vocabulary. We address this challenge with latent space planning, where we propose to explicitly leverage the constraints imposed by the conjugate relationships between states and actions in a learned plannable latent space. We evaluate both procedure planning and walkthrough planning on large-scale real-world instructional videos. Our experiments show that we are able to learn plannable semantic representations without explicit supervision. This enables sequential reasoning on real-world videos and leads to stronger generalization compared to existing planning approaches and neural network policies.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yezhou Yang
2. De-An Huang
3. Yi Yang
4. Luc Van Gool
5. Li Fei-Fei
6. Animesh Garg
7. Antonio Torralba
8. Zhiyuan Fang

Citations in the corpus

• Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, unknown): “Challenges include grounding keysteps [10, 36, 37, 89, 103, 104, 178, 207], procedural planning [15, 17, 22, 71, 143, 167, 196, 201], learning task structure [4, 9, 37, 107, 202, 205], and leveraging noisy narrations [89, 103, 104].”
• Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning (2020, background): “This topic has been explored for images and instructional videos in [9,4] respectively.”
• PlaTe: Visually-Grounded Planning With Transformers in Procedural Tasks (2021, methodology): “We consider a similar setup to [2]: given the start visual observation ot and a visual goal og that indicates the desired outcome for a particular task.”