Unity: A General Platform for Intelligent Agents

By Arthur Juliani, Vincent-Pierre Berges, Esh Vckay, Yuan Gao, Hunter Henry, Marwan Mattar, Danny B. Lange. 2018

970 citations recorded by the source corpus.

Recent advances in artificial intelligence have been driven by the presence of increasingly realistic and complex simulated environments. However, many of the existing environments provide either unrealistic visuals, inaccurate physics, low task complexity, restricted agent perspective, or a limited capacity for interaction among artificial agents. Furthermore, many platforms lack the ability to flexibly configure the simulation, making the simulated environment a black-box from the perspective of the learning system. In this work, we propose a novel taxonomy of existing simulation platforms and discuss the highest level class of general platforms which enable the development of learning environments that are rich in visual, physical, task, and social complexity. We argue that modern game engines are uniquely suited to act as general platforms and as a case study examine the Unity engine and open source Unity ML-Agents Toolkit. We then survey the research enabled by Unity and the Unity ML-Agents Toolkit, discussing the kinds of research a flexible, interactive and easily configurable general platform can facilitate.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Structured agents for physical construction (2019, methodology): “Our simulated task environment is a continuous, procedurally-generated 2D world implemented in Unity (Juliani et al., 2018) with the Box2D physics engine (Catto, 2013).”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, methodology): “Moreover, Unity Machine Learning Agents Toolkit implements core RL algorithms, games, simulations environments for training RL or IL based agents [150].”
A survey and critique of multiagent deep reinforcement learning (2018, background): “The Hanabi Learning Environment [309] was recently released24 and it is accompanied with a baseline (deep RL) agent [310]. Arena [311] is platform for multiagent research25 based on the Unity engine [312]. It has 35 multiagent games (e.g., social dilemmas) and supports communication among agents. It”
SAPIEN: A SimulAted Part-Based Interactive ENvironment (2020, background): “Arena [45], a platform that supports multi-agent environments, is built on top of Unity [23].”
IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks (2019, methodology): “Mujoco [3], Unity ML-Agents [36], PyBullet [7], and Dart [37] are some popular frameworks for physics simulation for reinforcement learning.”
Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation (2021, methodology): “This progress stems from the conﬂuence of strong algorithmic techniques [4–9] with accessible simulation software [10–14].”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, neutral): “Embodiedperceptionrequires visualperceptionandreasoning,understandingthe3-Drelations within a scene, and predicting and performing complex tasks based on visual information.”
Generalization of Reinforcement Learners with Working and Episodic Memory (2019, background): “, 2016), and new tasks built with the Unity 3D game engine (Juliani et al., 2018) that require the agent to 1) spot the difference between two scenes; 2) remember the location of a goal and navigate to it; or 3) infer an indirect transitive relation between objects.”
Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation (2020, background): “Various works have proposed generating synthetic data for tasks such as perception and planning in driving [55,18,53,14,48,68,2], indoor scene perception [70,75,46,57,59,24,4], game playing [6,28], robotic control [61] [6,63,56], optical flow estimation [7,58], home robotics [49,34,20] amongst many ”
Self-Supervised Exploration via Disagreement (2019, background): “, 2017), MNIST, Mujoco, Unity (Juliani et al., 2018) and a real robot.”
Multi-agent deep reinforcement learning: a survey (2021, background): “Many individual environments have been proposed which exhibit intricate structure and real-world complexity (Baker et al. 2020; Beattie et al. 2016; Johnson et al. 2016; Juliani et al. 2018; Song et al. 2019; Vinyals et al. 2017).”
Interactive Gibson Benchmark: A Benchmark for Interactive Navigation in Cluttered Environments (2019, background): “As a result, in recent years we have seen a new class of simulation environments [6, 7, 8, 9, 10, 11] that build upon renderers of synthetic or real-world scans.”
Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning (2019, methodology): “The Obstacle Tower environment uses the Unity platform and ML-Agents Toolkit (Juliani et al. 2018).”
ORRB - OpenAI Remote Rendering Backend (2019, background): “ORRB renderer is a standalone binary built with the Unity game engine.”
Embodied Question Answering in Photorealistic Environments With Point Cloud Perception (2019, background): “having realistic layouts and object occurrences), most are based in synthetic environments (on SUNCG [11] or Unity 3D models [12]) that are perceptually quite different from what agents embodied in the real world might experience.”
Reinforcement Learning Control of a Forestry Crane Manipulator (2021, methodology): “The Unity 3D simulation platform is used as a training environment, interfaced with the ML-Agents Toolkit [16] and the physics engine AGX Dynamics [17].”
AllenAct: A Framework for Embodied AI Research (2020, background): “ML-Agents [50] enables deﬁning new environments in the Unity game engine and provides a collection of example environments and RL algorithms.”
L2Explorer: A Lifelong Reinforcement Learning Assessment Environment (2022, methodology): “The project builds on Unity ML-Agents (Juliani et al. 2018), which exposes key parameters to create a reconfigurable, 3D, FPV reinforcement-learning environment.”
Deep Reinforcement Learning for Intelligent Transportation Systems: A Survey (2020, background): “A policy gradient-based deep RL method is proposed for adaptive traffic intersection control in [29], which presents experiments on a novel realistic traffic environment called Unity3D by using raw pixels as an input state to policy-based DQN.”