ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks

By Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke S. Zettlemoyer, Dieter Fox. 2019

1,140 citations recorded by the source corpus.

We present ALFRED (Action Learning From Realistic Environments and Directives), a benchmark for learning a mapping from natural language instructions and egocentric vision to sequences of actions for household tasks. ALFRED includes long, compositional tasks with non-reversible state changes to shrink the gap between research benchmarks and real-world applications. ALFRED consists of expert demonstrations in interactive visual environments for 25k natural language directives. These directives contain both high-level goals like “Rinse off a mug and place it in the coffee maker.” and low-level language instructions like “Walk to the coffee maker on the right.” ALFRED tasks are more complex in terms of sequence length, action space, and language than existing vision- and-language task datasets. We show that a baseline model based on recent embodied vision-and-language tasks performs poorly on ALFRED, suggesting that there is significant room for developing innovative grounded visual language understanding models with this benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

ReAct: Synergizing Reasoning and Acting in Language Models (2022, positive): “Table 9: An ReAct-IM prompt on the ALFWorld clean task.”
Experience Grounds Language (2020, background): “, 2019), a simulated house (Anderson et al., 2018; Thomason et al., 2019b; Shridhar et al., 2020), or the real”
Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, background): “Instructionfollowing models have also been of interest in the multi-modal learning literature (Fried et al., 2018; Shridhar et al., 2020; Min et al., 2022; Weir et al., 2022).”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, background): “Instructions in NLP. Previous work in NLP has studied “instructions” (Goldwasser and Roth, 2014) in different domains such as robotic instructions (Shridhar et al., 2020; Stepputtis et al., 2020), database commands (Kim et al., 2020), programming instructions (Lin et al., 2018; Shao and Nakashole,…”
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “These were developed to test various aspects of SayCan and were inspired by crowd sourcing via Amazon Mechanical Turk and in-person kitchen users, as well as benchmarks such as ALFRED [17] and BEHAVIOR [18].”
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, background): “These include: 1) generation of good search queries [22, 59] and reformulation [37, 51], 2) strategic exploration for navigating through the website [55, 56, 29], 3) robust language understanding for textual state and action spaces [3, 7, 17, 44], and 4) long-term”
Pre-Trained Language Models for Interactive Decision-Making (2022, neutral): “We note that LID is not limited to VirtualHome and BabyAI, but is straightforwardly applicable to other embodied environments, such as ALFRED [38] and iGibson [37].”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “Shridhar et al., 2020a;b; Kolve et al., 2017; Savva et al., 2019; Anderson et al., 2018).”
LILA: Language-Informed Latent Actions (2021, background): “On the other end of the spectrum are more recent approaches in end-to-end robot learning for more complicated tasks spanning navigation [44, 51] and manipulation [55].”
Language Models are Few-Shot Butlers (2021, methodology): “ALFWorld (Shridhar et al., 2021) is a goal-based textual environment mirroring the embodied ALFRED benchmark (Shridhar et al., 2020) with the TextWorld game engine (Côté et al., 2018).”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, methodology): “, 2018b), ALFRED (Shridhar et al., 2020), InteractiveQA (Gordon et al.”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, background): “Language instructions are a versatile way of defining goals, which is why they have been studied in the context of a variety of applications, such as instructions in grounded environments (Shridhar et al., 2020; Stepputtis et al., 2020; Min et al., 2022b; Weir et al., 2022) and database commands (Ki”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “The subfield of language models learning instruction-following capabilities aims to generate responses based on natural language commands, which have been extensively researched in language [64, 63, 3, 46], and multi-modality [59, 42] domains.”
Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “[57] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox.”
CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks (2022, methodology): “, 2018] and task completion [Shridhar et al., 2020], and may be feasible in a more task-agnostic CLiMB framework.”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “These embodied AI simulators also facilitate the collection of of task-based dataset [10, 11] which are tedious to collect in real-world as”
Skill Induction and Planning with Latent Language (2021, methodology): “Using an extremely reduced version of the AL-FRED household robotics dataset (Shridhar et al., 2020)—with 10% of labeled training instructions, no alignments during training, and no instructions at all during evaluation—our approach performs comparably a state-of-the-art model that makes much…”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, neutral): “More broadly, our work is also related to instruction-guided household task benchmarks such as ALFRED (Shridhar et al., 2020) and CHAI (Misra et al.”