Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion

By Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, G. Sukhatme. 2021

90 citations recorded by the source corpus.

Language-guided robots performing home and office tasks must navigate in and interact with the world. Grounding language instructions against visual observations and actions to take in an environment is an open challenge. We present Embodied BERT (EmBERT), a transformer-based model which can attend to high-dimensional, multi-modal inputs across long temporal horizons for language-conditioned task completion. Additionally, we bridge the gap between successful object-centric navigation models used for non-interactive agents and the language-guided visual task completion benchmark, ALFRED, by introducing object navigation targets for EmBERT training. We achieve competitive performance on the ALFRED benchmark, and EmBERT marks the first transformer-based model to successfully handle the long-horizon, dense, multi-modal histories of ALFRED, and the first ALFRED model to utilize object-centric navigation targets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, LLM Agents & Tool Use, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In LLM Agents & Tool Use

In Multimodal Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “Recent works have studied how to ground modern language models, by training them to accept additional environment inputs [30, 31, 32, 33, 34] or to directly output actions [35, 36, 37].”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, methodology): “, CLIP [71]) have been trained on joint image(s) and corresponding text captions via variants of a masked language modeling objective [72, 73, 74, 75], a contrastive loss [76, 77, 71] or other supervised objectives[78, 79].”
Pre-Trained Language Models for Interactive Decision-Making (2022, background): “This includes tasks that are not purely linguistic, and additionally require nontrivial planning and reasoning capabilities: for example, visionlanguage navigation (Majumdar et al., 2020; Fried et al., 2018; Suglia et al., 2021), instruction following (Zhang & Chai, 2021; Hill et al.”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “…power of neural architectures, various works looked into creating instruction-following agents that can perform manipulation (Lynch & Sermanet, 2020; 2021), navigation (Fried et al., 2018; Wang et al., 2019; Majumdar et al., 2020), or both (Suglia et al., 2021; Hill et al., 2020; Fu et al., 2019).”
Skill Induction and Planning with Latent Language (2021, methodology): “…(SL) 3 10% seq2seq seq2seq2seq S+(Shridhar et al., 2020) MOCA(Singh et al., 2020) Modular (Corona et al., 2021) ABP (Kim et al., 2021) EmBERT (Suglia et al., 2021) ET (Pashevich et al., 2021) HLSM(Blukis et al., 2021) * * HiTUT (Zhang and Chai, 2021) FILM (Min et al., 2021) Table 3: Detailed…”
The rise and potential of large language model based agents: a survey (2025, unknown): “The constraint is also found in some end-to-end models [254–256].”
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models (2022, methodology): “In navigation-only VLN datasets such as R2R [2], models that generate the action sequence end-to-end with a Transformer model can already achieve a good performance [37, 27].”
SQA3D: Situated Question Answering in 3D Scenes (2022, background): “…usually comprise multi-modal fusion (Tenenbaum & Freeman, 1996; Perez et al., 2018) and are trained from scratch (Wang et al., 2018; Fried et al., 2018; Wang et al., 2019), while recent efforts would employ pre-trained models (Pashevich et al., 2021; Hong et al., 2021a; Suglia et al., 2021).”
Do Embodied Agents Dream of Pixelated Sheep?: Embodied Decision Making using Language Guided World Modelling (2023, background): “, 2020) prompting many improvements to decision making conditioned on language instructions (Yu et al., 2018; Lynch & Sermanet, 2020; Nottingham et al., 2021; Suglia et al., 2021; Kuo et al., 2021; Zellers et al., 2021; Song et al., 2022a; Blukis et al., 2022).”