Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring

By Yichi Zhang, Joyce Yue Chai. 2021

92 citations recorded by the source corpus.

Despite recent progress, learning new tasks through language instructions remains an extremely challenging problem. On the ALFRED benchmark for task learning, the published state-of-the-art system only achieves a task success rate of less than 10% in an unseen environment, compared to the human performance of over 90%. To address this issue, this paper takes a closer look at task learning. In a departure from a widely applied end-to-end architecture, we decomposed task learning into three sub-problems: sub-goal planning, scene navigation, and object manipulation; and developed a model HiTUT (stands for Hierarchical Tasks via Unified Transformers) that addresses each sub-problem in a unified manner to learn a hierarchical task structure. On the ALFRED benchmark, HiTUT has achieved the best performance with a remarkably higher generalization ability. In the unseen environment, HiTUT achieves over 160% performance gain in success rate compared to the previous state of the art. The explicit representation of task structures also enables an in-depth understanding of the nature of the problem and the ability of the agent, which provides insight for future benchmark development and evaluation.

Subfields: LLM Agents & Tool Use, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In LLM Agents & Tool Use

In Robotics & Control

Citations in the corpus

PaLM-E: An Embodied Multimodal Language Model (2023, background): “Prior works focus on combining vision and language inputs in an embodied setting with the goal of direct action prediction (Guhur et al., 2022; Shridhar et al., 2022b;a; Zhang & Chai, 2021; Silva et al., 2021; Jang et al., 2022; Nair et al., 2022; Lynch et al., 2022; Brohan et al., 2022).”
Skill Induction and Planning with Latent Language (2021, methodology): “…Modular (Corona et al., 2021) ABP (Kim et al., 2021) EmBERT (Suglia et al., 2021) ET (Pashevich et al., 2021) HLSM(Blukis et al., 2021) * * HiTUT (Zhang and Chai, 2021) FILM (Min et al., 2021) Table 3: Detailed comparison of information available to models and baselines at training time and…”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “, 2021), subgoal prediction structures (SG) (Zhang and Chai, 2021; Blukis et al., 2021), or maintained voxel maps (Blukis et al.”
A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution (2021, methodology): “Modelling improvements have enabled these approaches to achieve good performance on complex navigation tasks [7, 9, 11, 13, 14, 37], a success that has not yet translated to mobile manipulation [12, 46, 47].”
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models (2022, background): “However, contemporary language-driven agents still require a large number of labeled examples (pairs of language instructions and gold trajectories) to learn each task, which is highly costly and hinders the development of truly versatile agents [34, 29, 25, 8, 37, 17, 40, 27, 11, 2, 16].”
FILM: Following Instructions in Language with Modular Methods (2021, methodology): “Most existing methods (Zhang & Chai, 2021; Kim et al., 2021; Nottingham et al., 2021) for EIF have relied on neural memory of various types (transformer embeddings, LSTM state), trained end-to-end with expert trajectories upon raw or pre-processed language/visual inputs.”