UnrealCV: Connecting Computer Vision to Unreal Engine

By Weichao Qiu, Alan L. Yuille. 2016

294 citations recorded by the source corpus.

Computer graphics can not only generate synthetic images and ground truth but it also offers the possibility of constructing virtual worlds in which: (i) an agent can perceive, navigate, and take actions guided by AI algorithms, (ii) properties of the worlds can be modified (e.g., material and reflectance), (iii) physical simulations can be performed, and (iv) algorithms can be learnt and evaluated. But creating realistic virtual worlds is not easy. The game industry, however, has spent a lot of effort creating 3D worlds, which a player can interact with. So researchers can build on these resources to create virtual worlds, provided we can access and modify the internal data structures of the games. To enable this we created an open-source plugin UnrealCV (this http URL) for a popular game engine Unreal Engine 4 (UE4). We show two applications: (i) a proof of concept image dataset, and (ii) linking Caffe with the virtual world to test deep network algorithms.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning from Simulated and Unsupervised Images through Adversarial Training (2016, background): “Human pose estimation with Kinect [35] and, more recently, a plethora of other tasks have been tackled using synthetic data [43, 42, 29, 34].”
Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks (2016, background): “Consequently, there has been a renewed interest in training models in the synthetic domain and applying them in real–world settings [8, 47, 38, 42, 25, 32, 35, 37].”
DeepMind Lab (2016, background): “This work is different and complementary to other recent projects which run as plugins to access internal content in the Unreal engine (Qiu and Yuille, 2016; Lerer et al., 2016).”
Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars (2017, background): “…Hattori et al. 2015), action recognition (Rahmani and Mian 2015, 2016; Roberto de et al. 2017), semantic segmentation (Richter et al. 2016), scene understanding (Handa et al. 2016b; Kohli et al. 2016; Qiu and Yuille 2016; Handa et al. 2016a), as well as in benchmark datasets (Handa et al. 2014).”
Conditional Generative Adversarial Network for Structured Domain Adaptation (2018, background): “Recent advances in computer graphics have stimulated the interests to train deep learning models on photo-realistic synthetic data with computergenerated annotations, and apply them in real-world settings [12, 29, 47, 37, 41, 48, 20, 19, 32, 22, 34, 33].”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, methodology): “Therefore, several methods for obtaining better object proposals have been proposed (Ren et al. (2015); Chen et al.”
UnrealCV: Virtual Worlds for Computer Vision (2017, background): “Initially, six virtual worlds are released for data generation and algorithm diagnosis[7, 10].”
SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth (2016, methodology): “Our process to generate the contents of the synthetic scenes observed has relied to the greatest degree possible on fully automatic methods, with object distributions statistically sampled from publicly available real-world scene repositories and randomly positioned within a physics simulation that…”
Meaningful maps with object-oriented semantic mapping (2016, unknown): “The proposed future research can be supported by recently published synthetic datasets [38] or high-ﬁdelity simulation environments [39].”
Few-Shot Adaptive Faster R-CNN (2019, background): “…data domain shift, which is important as learning from synthetic data is very promising way to address the lack of labeled training data [6, 38, 33]; Scenario-3 and Scenario-4 constructed with both real world collected datasets mainly aim for domain shift like illumination, camera condition,…”
Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data (2018, background): “Applications like optical flow [1], [3], scene flow [4], classification [18], stereo [5], [6], semantic segmentation [8], [9], [19], 3D keypoint extraction [20], object pose [11] and 3D reconstruction [2], [7] have all benefited from the use of synthetic training data.”
VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning (2019, methodology): “To this end, several plugins for game engines have been released, such as UETorch [27] and UnrealCV [39].”
Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization (2018, background): “The use of synthetic data for training and testing deep neural networks has gained in popularity in recent years, as evidenced by the availability of a large number of such datasets: Flying Chairs [4], FlyingThings3D [18], MPI Sintel [1], UnrealStereo [24, 36], SceneNet [9], SceneNet RGB-D [19], SYN”
Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training (2017, background): “The obstacles associated with developing a large dataset of real images can be circumvented by generating synthetic images [11]–[13].”
Synthetic Data for Deep Learning (2019, methodology): “[466, 467], an open-source plugin for the popular game engine Unreal Engine 4 that provides commands that allow to get and set camera location and field of view, get the set of objects in a scene together with their positions, set lighting parameters, modify properties of the objects such as materia”
Deep Nets: What have They Ever Done for Vision? (2018, background): “An advantage of synthetic data is that it enables us to generate, in principle, an inﬁnite amount of images and hence to systematically explore the eﬀect of varying factors like viewpoint and material properties, e.g., see Qiu and Yuille (2016); Al-corn et al. (2018).”
Procedural Modeling and Physically Based Rendering for Synthetic Data Generation in Automotive Applications (2017, methodology): “[25] who recently introduced the UnrealCV plugin, which enables generation of image- and accompanying ground truth annotations using Unreal Engine4 from Epic Games.”
Kubric: A scalable dataset generator (2022, unknown): “Playing4Data [78] (Game) × (Game) × × UnrealCV [75] UE4 × UE4 X × TDW [34] Unity × PhysX X × iGibson [106] PyRender × PyBullet X × Habitat [91] Magnum × Bullet X × OpenRooms [56] OptiX X – × × Omnidata [27] Blender X – × X Blenderproc [23] Blender X Bullet × × Kubric Blender X PyBullet X X”