Semantic Parsing for Text to 3D Scene Generation

By Angel X. Chang, Manolis Savva, Christopher D. Manning. 2014

39 citations recorded by the source corpus.

We propose text-to-scene generation as an application for semantic parsing. This is an application that grounds semantics in a virtual world that requires understanding of common, everyday language. In text to scene generation, the user provides a textual description and the system generates a 3D scene. For example, Figure 1 shows the generated scene for the input text “there is a room with a chair and a computer”. This is a challenging, open-ended problem that prior work has only addressed in a limited way. Most of the technical challenges in text to scene generation stem from the difficulty of mapping language to formal representations of visual scenes, as well as an overall absence of real world spatial knowledge from current NLP systems. These issues are partly due to the omission in natural language of many facts about the world. When people describe scenes in text, they typically specify only important, relevant information. Many common sense facts are unstated (e.g., chairs and desks are typically on the floor). Therefore, we focus on inferring implicit relations that are likely to hold even if they are not explicitly stated by the input text. Text to scene generation offers a rich, interactive environment for grounded language that is familiar to everyone. The entities are common, everyday objects, and the knowledge necessary to address this problem is of general use across many domains. We present a system that leverages user interactionwith 3D scenes to generate training data for semantic parsing approaches. Previous semantic parsing work has dealt with grounding text to physical attributes and relations (Matuszek et al., 2012; Krishnamurthy and Kollar, 2013), generating text for referring to objects (FitzGerald et al., 2013) and with connecting language to spatial relationships (Golland et al., 2010; Artzi and Zettlemoyer, 2013). Semantic parsing methods can also be applied to many aspects of text to scene generation. Furthermore, work on parsing instructions to robots (Matuszek et al., 2013; Tellex et al., 2014) has analogues in the context of discourse about physical scenes. In this extended abstract, we formalize the text to scene generation problem and describe it as a task for semantic parsing methods. To motivate this problem, we present a prototype system that incorporates simple spatial knowledge, and parses natural text to a semantic representation. By learning priors on spatial knowledge (e.g., typical positions of objects, and common spatial relations) our system addresses inference of implicit spatial constraints. The user can interactively manipulate the generated scene with textual commands, enabling us to refine and expand learned priors. Our current system uses deterministic rules to map text to a scene representation but we plan to explore training a semantic parser from data. We can leverage our system to collect user interactions for training data. Crowdsourcing is a promising avenue for obtaining a large scale dataset.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Information Extraction, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Information Extraction

In 3D Vision & Pose

Citations in the corpus

Visual Relationship Detection with Language Priors (2016, methodology): “Finally, they have been used to generate indoor images from sentences [39] and to improve image search [8,40].”
SceneFormer: Indoor Scene Generation with Transformers (2020, methodology): “Text inputs have been used to create detailed partial scenes [16, 6], such as a table with several small objects on it, and then insert this into the rest of the scene.”
Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation (2018, background): “…phrases detection and detection works and these works were mostly supposed to leveraging other tasks, such as object recognition [4,12,13,31,32,44,51,53,55], image classiﬁcation and retrieval [17,39], scene understanding and generation [2,3,18,23,24,61,65], as well as text grounding [27, 43, 49].”
Detecting Visual Relationships with Deep Relational Networks (2017, methodology): “They are essentially different from our work, which aims to provide a method dedicated to generic visual relationship detection.”