VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning

By Xiaofeng Gao, Ran Gong, Tianmin Shu, Xu Xie, Shu Wang, Song-Chun Zhu. 2019

67 citations recorded by the source corpus.

One of the main challenges of advancing task-oriented learning such as visual task planning and reinforcement learning is the lack of realistic and standardized environments for training and testing AI agents. Previously, researchers often relied on ad-hoc lab environments. There have been recent advances in virtual systems built with 3D physics engines and photo-realistic rendering for indoor and outdoor environments, but the embodied agents in those systems can only conduct simple interactions with the world (e.g., walking around, moving objects, etc.). Most of the existing systems also do not allow human participation in their simulated environments. In this work, we design and implement a virtual reality (VR) system, VRKitchen, with integrated functions which i) enable embodied agents powered by modern AI methods (e.g., planning, reinforcement learning, etc.) to perform complex tasks involving a wide range of fine-grained object manipulations in a realistic environment, and ii) allow human teachers to perform demonstrations to train agents (i.e., learning from demonstration). We also provide standardized evaluation benchmarks and data collection tools to facilitate a broad use in research on task-oriented learning and beyond.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “VRKitchen [22] is a virtual kitchen environment that is constructed using three modules: a physics-based and photorealistic kitchen environment which is constructed using Unreal Engine 4 (UE4), a user interface module which allows user to perform controls using a virtual reality device or a Python A”
SAPIEN: A SimulAted Part-Based Interactive ENvironment (2020, background): “Simulation environments that leverage game engines [52, 33, 3, 13, 25] come with manually designed or procedurally generated game scenes.”
ActioNet: An Interactive End-To-End Platform For Task-Based Data Collection And Augmentation In 3D Environment (2020, background): “VR Kitchen [6] is a 3D environment of virtual kitchens that allows the collection of human cooking demonstration dataset.”
iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks (2021, unknown): “VRKitchen [31] collected demon121 strations for five cooking tasks in simulation.”
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration (2020, background): “…et al., 2019; Xia et al., 2018; Brodeur et al., 2017; Zhu et al., 2017; Xia et al., 2018) , embodied question answering (Gordon et al., 2017; Wijmans et al., 2019; Das et al., 2018), or single agent task completion (Puig et al., 2018; Shridhar et al., 2019; Misra et al., 2018; Gao et al., 2019).”
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents (2021, background): “Beyond video game environments, many home environment simulators have been proposed recently (Brodeur et al., 2017; Savva et al., 2017; Puig et al., 2018; Yan et al., 2018; Gao et al., 2019; Savva et al., 2019) which offer visuallyrealistic scenes for evaluating Embodied AI.”
Learning Affordance Landscapes for Interaction Exploration in 3D Environments (2020, methodology): “Interaction in 3D environments Beyond navigation, recent work leverages simulated interactionbased environments [21, 29, 55, 45] to develop agents that can also perform actions (e.”
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense (2020, background): “[383] Xiaofeng Gao, Ran Gong, Tianmin Shu, Xu Xie, Shu Wang, SongChun Zhu, Vrkitchen: an interactive 3d virtual environment for taskoriented learning, arXiv preprint arXiv:1903.”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “Since kitchen scenes present a diverse set of object interactions in multi-step cooking activities, they are of great interest in this research domain [13, 23, 39].”
Interactive Gibson Benchmark: A Benchmark for Interactive Navigation in Cluttered Environments (2019, background): “AI2-THOR [6], VRKitchen [37], and VRGym [38] are based on a game engine.”
Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation (2020, background): “Various works have proposed generating synthetic data for tasks such as perception and planning in driving [55,18,53,14,48,68,2], indoor scene perception [70,75,46,57,59,24,4], game playing [6,28], robotic control [61] [6,63,56], optical flow estimation [7,58], home robotics [49,34,20] amongst many ”