Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications

By Matthias Müller, Vincent Casser, Jean Lahoud, Neil G. Smith, Bernard Ghanem. 2017

203 citations recorded by the source corpus.

We present a photo-realistic training and evaluation simulator (Sim4CV) (http://www.sim4cv.org) with extensive applications across various fields of computer vision. Built on top of the Unreal Engine, the simulator integrates full featured physics based cars, unmanned aerial vehicles (UAVs), and animated human actors in diverse urban and suburban 3D environments. We demonstrate the versatility of the simulator with two case studies: autonomous UAV-based tracking of moving objects and autonomous driving using supervised learning. The simulator fully integrates both several state-of-the-art tracking algorithms with a benchmark evaluation tool and a deep neural network architecture for training vehicles to drive autonomously. It generates synthetic photo-realistic datasets with automatic ground truth annotations to easily extend existing real-world datasets and provides extensive synthetic data variety through its ability to reconfigure synthetic worlds on the fly using an automatic world generation tool.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

SAPIEN: A SimulAted Part-Based Interactive ENvironment (2020, background): “To allow for more interactive features in environments, researchers leverage partial functionalities of game engines or physics engine to provide photorealistic rendering together with interactions [54, 34, 25, 38, 3, 59, 13, 26].”
Auxiliary Tasks in Multi-task Learning (2018, methodology): “Following publications proposed methods for the fully-automatic generation of training data from video games [15, 25] and custom simulators based on game engines [23] or procedural models [32].”
Conditional Affordance Learning for Driving in Urban Environments (2018, background): “It is well known that for imitation learning utilizing a single ﬁxed camera pose leads to unstable behavior [20, 38] due to control errors and unexpected observations reinforcing each other.”
MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning (2021, background): “The simulators GTA V [28], Sim4CV [31], AIRSIM [47], CARLA [10] and its derived project SUMMIT [4], MACAD [34] realistically preserve the delicate appearance and transition of the physical world, such as lighting conditions, weather system, even the time of a day shift.”
A Survey on 3D Object Detection Methods for Autonomous Driving Applications (2019, methodology): “Another simulation tool, Sim4CV [36] allows easy environment customization and simultaneous multi-view rendering of the driving scenes, while providing ground-truth bounding boxes for object detection”
Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization (2018, methodology): “ single image. A promising approach to overcome this limitation is to use a graphic simulator to generate automatically labeled data. Several such simulated datasets have been created in recent years [1,9,4,18,24,36,19,27,26,7,21,33]. For the most part, these datasets are expensive to generate, requ”
Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects (2018, methodology): “hreaded sequential frame grabbing, the plugin generates data at a rate of 50–100 Hz, which is signiﬁcantly faster than either the default UE4 screenshot function or the publicly available Sim4CV tool [19]. Fig.1shows examples of images generated using our plugin, illustrating the diversity and varie”
Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data (2018, background): “[11] built 2019 International Conference on Robotics and Automation (ICRA) Palais des congres de Montreal, Montreal, Canada, May 20-24, 2019”
Teaching UAVs to Race: End-to-End Regression of Agile Controls in Simulation (2017, methodology): “Recent work [6, 8, 27, 37, 40] highlights how modern game engines can be used to generate photo-realistic training datasets and pixel-accurate segmentation masks.”