Horizon: Facebook's Open Source Applied Reinforcement Learning Platform

By Jason Gauci, Edoardo Conti, Yitao Liang, Kittipat Virochsiri, Yuchen He, Zachary Kaden, Vivek Narayanan, Xiaohui Ye, Scott Fujimoto. 2018

147 citations recorded by the source corpus.

In this paper we present Horizon, Facebook's open source applied reinforcement learning (RL) platform. Horizon is an end-to-end platform designed to solve industry applied RL problems where datasets are large (millions to billions of observations), the feedback loop is slow (vs. a simulator), and experiments must be done with care because they don't run in a simulator. Unlike other RL platforms, which are often designed for fast prototyping and experimentation, Horizon is designed with production use cases as top of mind. The platform contains workflows to train popular deep RL algorithms and includes data preprocessing, feature transformation, distributed training, counterfactual policy evaluation, optimized serving, and a model-based data understanding tool. We also showcase and describe real examples where reinforcement learning models trained with Horizon significantly outperformed and replaced supervised learning systems at Facebook.

Subfields: Artificial Intelligence, Automotive Engineering, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Towards Characterizing Divergence in Deep Q-Learning (2019, methodology): “DQL algorithms have been applied fruitfully in video games (Mnih et al., 2015), robotics (Kalashnikov et al., 2018; Haarnoja et al., 2018a), and user interactions on social media (Gauci et al., 2018).”
A Minimalist Approach to Offline Reinforcement Learning (2021, background): “This questions the empirical effectiveness of offline RL for safety-critical real-world use cases [Mandel et al., 2014, Gottesman et al., 2018, Gauci et al., 2018, Jaques et al., 2019, Matsushima et al., 2020] as well as the current trend of reporting only the mean-value of the final policy in…”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “ReAgent (Gauci et al., 2018) is primarily aimed at offline/batch RL from large datasets in production settings.”
rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch (2019, background): “Facebook Horizon [33] offers a subset of algorithms and focuses on applications toward production at scale.”
Benchmarking Batch Deep Reinforcement Learning Algorithms (2019, methodology): “This setup is used to closely match batch settings used by real-world systems, which generally rely on a single behavioral for a ﬁxed period of time [Gauci et al., 2018].”
Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, background): “[19] Jason Gauci, Edoardo Conti, Yitao Liang, Kittipat Virochsiri, Zhengxing Chen, Yuchen He, Zachary Kaden, Vivek Narayanan, and Xiaohui Ye.”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, methodology): “The Horizon platform (Gauci et al. 2018) and Decision Service (Agarwal et al.”
Idiosyncrasies and challenges of data driven learning in electronic trading (2018, unknown): “, 2017], Horizon [Gauci et al., 2018], dopamine [Bellemare et al.”
BoTorch: Programmable Bayesian Optimization in PyTorch (2019, methodology): “Our experiment uses the Cartpole environment from OpenAI gym (Brockman et al., 2016) and the default DQN agent implemented in Horizon (Gauci et al., 2018).”
Stable-Baselines3: Reliable Reinforcement Learning Implementations (2021, methodology): “Only a few RL libraries oﬀer more than a brief API documentation (garage contributors, 2019; Liang et al., 2018; Kuhnle et al., 2017; Guadarrama et al., 2018), and some are notoriously hard to understand.”
Harms from Increasingly Agentic Algorithmic Systems (2023, background): “Major recommendation companies such as Meta [Gauci et al. 2019], YouTube [Association for Computing Machinery (ACM) 2019], and Spotify [Engineering 2021] have already deployed RL-based recommender systems on their live products.”
CAQL: Continuous Action Q-Learning (2019, background): “, 2013) and recommender systems (RSs) (Gauci et al., 2018).”
An empirical investigation of the challenges of real-world reinforcement learning (2020, methodology): “The Horizon platform [Gauci et al. 2018] and Decision Service [Agarwal et al.”
An Introduction to Deep Reinforcement Learning (2018, unknown): “For example, Gauci et al. (2018) describe how Facebook uses Deep RL such as for pushing notifications and for faster video loading with smart prefetching.”
Lessons from Real-World Reinforcement Learning in a Customer Support Bot (2019, background): “Horizon is based on the same principles as the Decision Service but has added the ﬂexibility of optimizing any kind of ONNX model.”
RecSim: A Configurable Simulation Platform for Recommender Systems (2019, methodology): “Other frameworks also provide standard RL algorithms with libraries integrated with OpenAI Gym [Gauci et al., 2018, Guadarrama et al., 2018].”
Reinforcement Learning based Recommender Systems: A Survey (2021, background): “Although some environment simulators have recently been developed for RLRSs [153, 154, 155], we believe that there is an essential need to have a powerful simulator, preferably a strong unified evaluation framework, for RLRSs, and the RS field in general.”