Horde: a scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction

By Richard S. Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M. Pilarski, Adam White, Doina Precup. 2011

543 citations recorded by the source corpus.

Maintaining accurate world knowledge in a complex and changing environment is a perennial problem for robots and other artificial intelligence systems. Our architecture for addressing this problem, called Horde, consists of a large number of independent reinforcement learning sub-agents, or demons. Each demon is responsible for answering a single predictive or goal-oriented question about the world, thereby contributing in a factored, modular way to the system's overall knowledge. The questions are in the form of a value function, but each demon has its own policy, reward function, termination function, and terminal-reward function unrelated to those of the base problem. Learning proceeds in parallel by all demons simultaneously so as to extract the maximal training information from whatever actions are taken by the system as a whole. Gradient-based temporal-difference learning methods are used to learn efficiently and reliably with function approximation in this off-policy setting. Horde runs in constant time and memory per time step, and is thus suitable for learning online in real-time applications such as robotics. We present results using Horde on a multi-sensored mobile robot to successfully learn goal-oriented behaviors and long-term predictions from off-policy experience. Horde is a significant incremental step towards a real-time architecture for efficient learning of general knowledge from unsupervised sensorimotor interaction.

Subfields: Aerospace Engineering, Artificial Intelligence, Social Psychology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Social Psychology

In Reinforcement Learning

Citations in the corpus

The Arcade Learning Environment: An Evaluation Platform for General Agents (2012, background): “Similar ideas have been developed around the theme of lifelong learning : learning a reusable, highlevel understanding of the world from raw sensory data (Thrun & Mitchell, 1995; Pierce & Kuipers, 1997; Stober & Kuipers, 2008; Sutton et al., 2011).”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Finally, it is often useful to learn multiple general value functions (Sutton et al., 2011; Mankowitz et al., 2018; Lample & Chaplot, 2016; Mirowski et al., 2017; Jaderberg et al., 2017b) or options (Sutton et al.”
The Predictron: End-To-End Learning and Planning (2016, methodology): “In the vector case, these are known as general value functions (Sutton et al., 2011).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, neutral): “e solving prediction problems, with the former predicting future returns, and the latter predicting future states. Moreover, model-free methods can be modiﬁed to predict even more general quantities (Sutton et al., 2011), such as return values with different discount factors, return values for diffe”
Independently Controllable Features (2017, positive): “Our approach is similar in spirit to the Horde architecture (Sutton, 2011).”
Gradient Episodic Memory for Continual Learning (2017, neutral): “The use of task descriptors is similar in spirit to recent work in Reinforcement Learning [Sutton et al., 2011, Schaul et al., 2015], where task or goal descriptors are also fed as input to the system.”
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, background): “Recently, value functions have also been generalized as V (s, g) in order to represent the utility of state s for achieving a given goal g ∈ G [33, 21].”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “The answer to a GVF question is defined as the value function, v : H → R, which gives the expected cumulative discounted cumulant from any history defined as: v(ht) = E[c(Ht+1) + γ(Ht+1)v(Ht+1)|Ht = ht, At+1 ∼ π(.”
Efficient Lifelong Learning with A-GEM (2018, methodology): “More generally, tasks descriptors have been used in Reinforcement Learning with similar motivations by several others (Sutton et al., 2011; Schaul et al., 2015; Baroni et al., 2017), and it is also a key ingredient in all the zero/few-shot learning algorithms (Lampert et al., 2014; Xian et al.,…”
Deep Reinforcement Learning and the Deadly Triad (2018, background): “An off-policy learning algorithm can estimate values for a policy that differs from the actual behaviour, which allows an agent to learn about policies other than the one it follows and to learn about many different policies in parallel (Sutton et al., 2011).”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, background): “Most importantly, off-policy methods allow an agent to learn about many different policies at once, forming the basis for a predictive understanding of an agent’s environment [Sutton et al., 2011, White, 2015] and enabling the learning of options [Sutton et al., 1999, Precup, 2000].”
Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation (2018, neutral): “4c shows GC-DQL GC-DQL-sep CAPs (ours) (a) Birds-eye view for each approach of the resulting trajectories when attempting to go to either the red, green, or blue goal location (five attempts per goal location).”
Learning to Act by Predicting the Future (2016, background): “Predictive representations in the form of generalized value functions were advocated by Sutton et al. (2011).”
Linear Off-Policy Actor-Critic (2012, background): “…methods are able to, for example, learn about an optimal policy while executing an exploratory policy (Sutton & Barto, 1998), learn from demonstration (Smart & Kaelbling, 2002), and learn multiple tasks in parallel from a single sensorimotor interaction with an environment (Sutton et al., 2011).”
Increasing the Action Gap: New Operators for Reinforcement Learning (2015, background): “…and may be crucial for systems operating at a ﬁne time scale such as video games (Togelius et al. 2009; Bellemare et al. 2013), real-time markets (Jiang and Powell 2015), and robotic platforms (Riedmiller et al. 2009; Hoburg and Tedrake 2009; Deisenroth and Rasmussen 2011; Sutton et al. 2011).”
A Distributional Perspective on Reinforcement Learning (2017, background): “A recurring theme in artiﬁcial intelligence is the idea of an agent learning from a multitude of predictions (Caruana 1997; Utgoff & Stracuzzi 2002; Sutton et al. 2011; Jaderberg et al. 2017).”