Learning Interpretable Spatial Operations in a Rich 3D Blocks World

By Yonatan Bisk, Kevin J. Shih, Yejin Choi, Daniel Marcu. 2017

65 citations recorded by the source corpus.

In this paper, we study the problem of mapping natural language instructions to complex spatial actions in a 3D blocks world. We first introduce a new dataset that pairs complex 3D spatial operations to rich natural language descriptions that require complex spatial and pragmatic interpretations such as “mirroring”, “twisting”, and “balancing”. This dataset, built on the simulation environment of Bisk, Yuret, and Marcu (2016), attains language that is significantly richer and more complex, while also doubling the size of the original dataset in the 2D environment with 100 new world configurations and 250,000 tokens. In addition, we propose a new neural architecture that achieves competitive results while automatically discovering an inventory of interpretable spatial operations (Figure 5).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Experience Grounds Language (2020, background): “In general, simple worlds do not entail simple concepts: even in block worlds (Bisk et al., 2018) where concepts like “mirroring”—one of a massive set of physical phenomena that we generalize and apply to abstract concepts—appear.”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “There have also been numerous visual reasoning models (Hudson & Manning, 2018; Santoro et al., 2017; Hu et al., 2017; Perez et al., 2018; Zhu et al., 2017; Mascharka et al., 2018; Suarez et al., 2018; Cao et al., 2018; Bisk et al., 2018; Misra et al., 2018; Aditya et al., 2018).”
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding (2018, background): “From a broader perspective, Misra et al. [2017] explored learning to reason by asking questions and Bisk et al. [2018] studied spatial reasoning in a 3D blocks world.”
RAVEN: A Dataset for Relational and Analogical Visual REasoNing (2019, background): “[4] studied visual reasoning in a 3D block world.”
Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation (2019, background): “In Vision-and-Language Navigation (VLN) tasks, agents must follow natural language navigation instructions through either simulated (Macmahon et al., 2006; Yan et al., 2018; Bisk et al., 2018; Shah et al., 2018), simulations of realistic (Blukis et al.”
Prospection: Interpretable plans from language by predicting the future (2019, methodology): “The model is trained on supervised demonstration data collected from an expert policy as per previous work [15], [4].”
Why Build an Assistant in Minecraft? (2019, background): “In [13, 12], the authors explore natural language instruction for block placement, first in 2D and then in 3D.”
Interactive Language: Talking to Robots in Real Time (2022, unknown): “Hindsight Language Labeling BLOCKS [50], [51] 30 n/r 7 7 3”
Vision-Based Navigation With Language-Based Assistance via Imitation Learning With Indirect Intervention (2018, background): “Simple simulators as Puddle World Navigation [31] and Rich 3D Blocks World [5, 4] have facilitated understanding of fundamental representational and grounding issues by allowing for fast experimentation in easily-managed environments.”
MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks (2021, background): “Unlike prior work in situated collaborative tasks (Liu et al., 2013; Bisk et al., 2018; Suhr et al., 2019), a key focus of our work is to facilitate theory of mind modeling—the a r X i v : 2109 .”