Multi-timescale nexting in a reinforcement learning robot

By Joseph Modayil, Adam White, Richard S. Sutton. 2011

133 citations recorded by the source corpus.

The term ‘nexting’ has been used by psychologists to refer to the propensity of people and many other animals to continually predict what will happen next in an immediate, local, and personal sense. The ability to ‘next’ constitutes a basic kind of awareness and knowledge of one’s environment. In this paper we present results with a robot that learns to next in real time, making thousands of predictions about sensory input signals at timescales from 0.1 to 8 seconds. Our predictions are formulated as a generalization of the value functions commonly used in reinforcement learning, where now an arbitrary function of the sensory input signals is used as a pseudo reward, and the discount rate determines the timescale. We show that six thousand predictions, each computed as a function of six thousand features of the state, can be learned and updated online ten times per second on a laptop computer, using the standard temporal-difference(λ) algorithm with linear function approximation. This approach is sufficiently computationally efficient to be used for real-time learning on the robot and sufficiently data efficient to achieve substantial accuracy within 30 minutes. Moreover, a single tile-coded feature representation suffices to accurately predict many different signals over a significant range of timescales. We also extend nexting beyond simple timescales by letting the discount rate be a function of the state and show that nexting predictions of this more general form can also be learned with substantial accuracy. General nexting provides a simple yet powerful mechanism for a robot to acquire predictive knowledge of the dynamics of its environment.

Subfields: Artificial Intelligence, Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

Citations in the corpus

Successor Features for Transfer in Reinforcement Learning (2016, background): “’s [16] hypothesis that relevant knowledge about the world can be expressed in the form of many predictions naturally translates to SFs: if φ is expressive enough, the agent should be able to represent any relevant reward function.”
Reward is enough (2021, background): “of future observations [31]) may provide effective principles for understanding experience, but do not provide a principle for action selection and therefore cannot in isolation be enough for goal-oriented intelligence.”
Distributed Policy Evaluation Under Multiple Behavior Strategies (2013, background): “The predictions by the agents are made in the form of value functions [2], [6], [7].”
A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning (2016, background): “The generalization to state-based γ and λ have not yet been widely considered, though the concept was introduced more than a decade ago [22, 27] and the generalization shown to be useful [22, 10, 14, 26].”
Incremental Off-policy Reinforcement Learning Algorithms (2017, background): “Modayil et al. (2014) hypothesized that, as with animal learning, a knowledge representation could be developed by grounding them to the predictions of the most primitive sensorimotor facts of the agent experience.”
Universal Value Function Approximators (2015, background): “For example, the Horde architecture (Sutton et al., 2011) consists of a discrete set of value functions ('demons'), all of which may be learnt simultaneously from a single stream of experience, by bootstrapping off-policy from successive value estimates (Modayil et al., 2014).”
True Online Temporal-Difference Learning (2015, background): “More detail on the subject and experimental setting can be found in Hebert et al. (2014). general value functions and nexting; Sutton et al., 2011; Modayil et al., 2014).”
True Online Emphatic TD(λ): Quick Reference and Implementation Guide (2015, methodology): “Here we present these ideas without assuming access to an underlying state (as in Modayil, White & Sutton 2014).”
Surprise and Curiosity for Big Data Robotics (2014, methodology): “Beyond the temporal scale of the problem, off-policy life-long learning enables additional scaling in the number of things that can be learned in parallel, as demonstrated by recent predictive, learning systems (see Modayil et al 2012, White et al 2013).”