FILM: Following Instructions in Language with Modular Methods

By So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, Ruslan Salakhutdinov. 2021

196 citations recorded by the source corpus.

Recent methods for embodied instruction following are typically trained end-to-end using imitation learning. This often requires the use of expert trajectories and low-level language instructions. Such approaches assume that neural states will integrate multimodal semantics to perform state tracking, building spatial memory, exploration, and long-term planning. In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene and (2) performs exploration with a semantic search policy, to achieve the natural language goal. Our modular method achieves SOTA performance (24.46 %) with a substantial (8.17 % absolute) gap from previous work while using less data by eschewing both expert trajectories and low-level instructions. Leveraging low-level language, however, can further increase our performance (26.49 %). Our findings suggest that an explicit spatial memory and a semantic search policy can provide a stronger and more general representation for state-tracking and guidance, even in the absence of expert trajectories or low-level instructions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, LLM Agents & Tool Use, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In LLM Agents & Tool Use

In Multimodal Learning

Citations in the corpus

Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, background): “…are a versatile way of deﬁn-ing goals, which is why they have been studied in the context of a variety of applications, such as instructions in grounded environments (Shrid-har et al., 2020; Stepputtis et al., 2020; Min et al., 2022b; Weir et al., 2022) and database commands (Kim et al., 2020).”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “The subfield of language models learning instruction-following capabilities aims to generate responses based on natural language commands, which have been extensively researched in language [64, 63, 3, 46], and multi-modality [59, 42] domains.”
Skill Induction and Planning with Latent Language (2021, methodology): “…(Corona et al., 2021) ABP (Kim et al., 2021) EmBERT (Suglia et al., 2021) ET (Pashevich et al., 2021) HLSM(Blukis et al., 2021) * * HiTUT (Zhang and Chai, 2021) FILM (Min et al., 2021) Table 3: Detailed comparison of information available to models and baselines at training time and inference.”
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models (2022, methodology): “We compare with two main baseline models, HLSM [3] and FILM [24].”
Benchmarking Generalization via In-Context Instructions on 1, 600+ Language Tasks (2022, background): “Language “instructions” are a versatile way of defining goals, which is why they have been studied in the context of a variety of applications, such as instructions in grounded environments (Shridhar et al., 2020; Stepputtis et al., 2020; Min et al., 2022b; Weir et al., 2022) and database commands (”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, unknown): “Thus, it is crucial for embodied agents to align and generalize their abilities from the cyber space to the physical world.”
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model (2023, background): “Another example using the OCR expert from DocVQA [44] is presented in Fig.”
Online Continual Learning For Interactive Instruction Following Agents (2024, methodology): “Most previous methods (Singh et al., 2021; Pashevich et al., 2021; Min et al., 2022; Kim et al., 2023) for object localization utilize a two-stage approach, separating it into object class prediction and mask generation to enhance object localization.”