Neural Program Synthesis from Diverse Demonstration Videos

By Shao-Hua Sun, Hyeonwoo Noh, Sriram Somasundaram, Joseph J. Lim. 2018

81 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Code Generation & Program Synthesis

Citations in the corpus

Reasoning Like Program Executors (2022, background): “Other lines of research leverage program execution in inference as a reliable sanity guarantee for generated programs by pruning non-executable candidates (Wang et al., 2018b; Chen et al., 2019b, 2021; Odena et al., 2020; Ellis et al., 2019; Chen et al., 2019b; Sun et al., 2018; Zohar and Wolf, 2018”
Neural Logic Machines (2019, neutral): “…program induction and synthesis (Neelakantan et al., 2016; Reed & De Freitas, 2016; Kaiser & Sutskever, 2016; Parisotto et al., 2017; Devlin et al., 2017; Bunel et al., 2018; Sun et al., 2018) are recently introduced to solve problems by synthesizing computer programs with neural augmentations.”
Modeling question asking using neural program generation (2019, background): “Our work also builds on neural network approaches to program synthesis, which have been applied to many different domains [4, 24, 25].”
Execution-Guided Neural Program Synthesis (2018, background): “For example, Sun et al. (2018) propose to synthesize the program from demonstration videos, which can be viewed as sequences of states.”
Learning Compositional Neural Programs with Recursive Tree Search and Planning (2019, background): “In [Sun et al., 2018], programs are generated from visual demonstrations.”
Synthesizing Environment-Aware Activities via Activity Sketches (2019, unknown): “A recent body of work has developed neural approaches to program generation using user-provided examples [8, 4], visual demonstrations [17], and descriptions [6].”
LEGO: Latent Execution-Guided Reasoning for Multi-Hop Question Answering on Knowledge Graphs (2021, methodology): “…iteratively retrieves a subgraph from the KG starting from the topic entities and obtains answers by ranking entities on the subgraph by graph nets (Sun et al., 2018a); (2) EmbedKGQA (Sax-ena et al., 2020), which learns a score function between the question embeddings and the entity embeddings…”
CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning (2022, background): “For instance, in [Sun et al., 2018], programming language contains basic functions for object perception, action, and control flows.”
IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks (2019, background): “Domain Knowledge : Instruction manuals [45, 1], programs [26, 46], and video demonstrations [21] can be used as domain knowledge to help solving complex tasks.”
SpreadsheetCoder: Formula Prediction from Semi-structured Context (2021, background): “Some existing works on program synthesis have evaluated the semantic equivalence based on the execution results (Devlin et al., 2017; Bunel et al., 2018; Sun et al., 2018).”
Discovering Motor Programs by Recomposing Demonstrations (2020, background): “A few recent works have striven to do so, by representing these tasks as programs (Xu et al., 2018; Sun et al., 2018), or as task graphs (Huang et al.”
Learning to Synthesize Programs as Interpretable and Generalizable Policies (2021, methodology): “On the other hand, program synthesis works that aim to represent desired behaviors using flexible domain-specific programs often require extra supervision such as input/output pairs [16–20] or expert demonstrations [21, 22], which can be difficult to obtain.”
CompILE: Compositional Imitation Learning and Execution (2018, background): “A related recent approach decomposes demonstration sequences into underlying programs (Sun et al., 2018) in a fully-supervised setting, based on a seq2seq (Sutskever et al., 2014; Vinyals et al., 2015) model without explicitly modeling segmentation.”