Grounding Language in Play

By Corey Lynch, Pierre Sermanet. 2020

80 citations recorded by the source corpus.

Subfields: Natural Language Processing, LLM Agents & Tool Use

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In LLM Agents & Tool Use

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “In BC NL we feed the full instruction i into the policy – this approach is representative of standard RL or BC-based instruction following methods [13, 20, 21, 22].”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “At the same time, by leveraging the better representation power of neural architectures, a number of works have looked into creating instruction-following agents that can perform manipulation [29, 30], navigation [11, 54, 31], or both [49, 16, 12].”
LILA: Language-Informed Latent Actions (2021, background): “…sought to combine language and robotics, spanning approaches that map language to planning primitives [9, 30, 31, 32, 33], perform imitation learning from demonstrations and instructions [11, 34, 35, 36], and pair language instructions with reward functions for reinforcement learning [12, 37, 38].”
Imitating Interactive Intelligence (2020, methodology): “…the same spirit as recent work aimed to teach virtual or physical robots to follow instructions provided in natural language (Hermann et al., 2017; Lynch and Sermanet, 2020) but attempts to go beyond it by emphasising the interactive and language production capabilities of the agents we develop.”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “Embodied AI (EAI) agents take in multi-sensory input, such as vision and language [22] or vision and audio [16] and take actions such as navigation [3], object manipulation [48, 51], or both [30, 66], in a simulated environment.”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “al [48] proposed an end-to-end system for grounding language in continuous control, but it requires several hours of human teleoperation data for a single simulated desk setting.”
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “A direction to address this limitation is to relabel the dataset with a variety of human-provided annotations [24], which could enable the system to handle more variability in the language structure.”
Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning (2020, methodology): “See [38, 41] for a complete discussion of the 18 tasks, which span different families, e.”
Grounding Language to Autonomously-Acquired Skills via Goal Generation (2021, unknown): “Closer to us, Lynch & Sermanet (2020) also decouple skill learning from language acquisition in a goal-conditioned imitation learning paradigm by mapping both language goals and images goals to a shared representation space.”
ELLA: Exploration through Learned Language Abstraction (2021, background): “Even simple instruction following problems are difﬁcult to learn [29], which has made 2D environments important test-beds for new learning methods [3, 11, 47].”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, methodology): “Recently, a number of end-to-end approaches [13, 12, 61] have been proposed for conditioning BC agents with language instructions.”
R3M: A Universal Visual Representation for Robot Manipulation (2022, background): “Prior works have explored the use of natural language in robot manipulation, primarily as a means of task specification [51, 52, 35, 53] or reward learning [54].”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, unknown): “The project website is robotics-transformer-x.github.io.”
PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards (2020, background): “[23] learn a goal-image- and language-conditioned policy, using behavior cloning.”
Transformers for One-Shot Visual Imitation (2020, background): “There are myriad ways to introduce goal conditioning, including with the robot’s state [24], ”goal” images of the ﬁnal state [25, 26, 27], natural language [28], and video or images of humans [29, 30].”
Episodic Transformer for Vision-and-Language Navigation (2021, methodology): “Instruction types include structured commands or logic programs [22,43,53], natural language [12,61], target state images [37], or a mix [38].”
PROST: Physical Reasoning about Objects through Space and Time (2021, background): “Additionally, other work has explored grounding language in simulations or the real world (Hill et al., 2020; Lynch and Sermanet, 2020).”
3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (2024, unknown): “To better handle action multimodality, other approaches discretize action dimensions and use cross entropy losses [45, 69, 84].”
Language Grounding with 3D Objects (2021, background): “Agents can perform information-seeking behaviors on objects to improve their world [15, 16, 17] and language understanding [18, 19, 20].”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, methodology): “Furthermore, we compare performance with and without using the pre-trained language model, and observe that without the pre-trained language model performance is worse on seen instructions and drops significantly more on unseen instructions (up to 23% vs 10%), suggesting that the pre-trained languag”