Unifying Task Specification in Reinforcement Learning

By Martha White. 2016

97 citations recorded by the source corpus.

Reinforcement learning tasks are typically specified as Markov decision processes. This formalism has been highly successful, though specifications often couple the dynamics of the environment and the learning objective. This lack of modularity can complicate generalization of the task specification, as well as obfuscate connections between different task settings, such as episodic and continuing. In this work, we introduce the RL task formalism, that provides a unification through simple constructs including a generalization to transition-based discounting. Through a series of examples, we demonstrate the generality and utility of this formalism. Finally, we extend standard learning constructs, including Bellman operators, and extend some seminal theoretical results, including approximation errors bounds. Overall, we provide a well-understood and sound formalism on which to build theoretical results and simplify algorithm use and development.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Generalized Off-Policy Actor-Critic (2019, methodology): “= ∑ s dπ(s)i(s)vπ(s), (4) which is equivalent to optimizing the average reward if both γ and i are constant (White, 2017).”
Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift (2019, neutral): “This is analogous to the perspective of the discount factor as a probability of terminating (White 2017), and is related to the constraint that arises in the dual formulation of the value function (Wang et al.”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, methodology): “(1) = Rt+1 + t+1Gt+1 We use transition-based discounting : S ⇥ A ⇥ S ! [0, 1], as it unifies continuing and episodic tasks [White, 2017].”
The Utility of Sparse Representations for Control in Reinforcement Learning (2018, methodology): “The environment is formalized by a Markov decision process (MDP), with states S , actions A , transition probabilities Pr : S × A × S → [0 , 1] , rewards R : S ×A×S → R and discount function γ : S ×A×S → [0 , 1] (White 2017).”
Incremental Off-policy Reinforcement Learning Algorithms (2017, background): “Chapter 9 and 10 are based on a published journal paper (Sutton, Mahmood & White 2016).”
Combined Reinforcement Learning via Abstract Representations (2018, background): “…be extended to the stochastic case as discussed in Section 6); (iv) the environment’s reward function R : S × A → R where R is a continuous set of possible rewards in a range R max ∈ R + (e.g., [0 , R max ] ); and (v) a general discount factor G : S × A × S → [0 , 1) , similarly to White (2016) 1 .”
Value Propagation Networks (2018, background): “The second output given by Φ is a single value pi,j , which represent a propagation parameter associated to the position, or simply a state-dependent discount factor (White, 2016): pi,j ≈ 1 means that the neighboring values vi′,j′ propagate through i, j, while pi,j ≈ 0 means that position i, j is bl”
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation (2019, background): “We refer the reader to White (2017) for more usage of the interest function.”
Organizing Experience: a Deeper Look at Replay Mechanisms for Sample-Based Planning in Continuous State Domains (2018, background): “The generalization to a discount function γ allows for a unified specification of episodic and continuing tasks [White, 2017], both of which are considered in this work.”
Optimizing for the Future in Non-Stationary MDPs (2020, background): “Then a single, never-ending, episode in that (PO)MDP can be considered with state dependent discounting (White, 2017) for task specifications.”
Two-Timescale Networks for Nonlinear Value Function Approximation (2019, background): “We assume the agents act in a finite Markov Decision Process (MDP), with notation from (White, 2017).”
Time Limits in Reinforcement Learning (2017, methodology): “Related to the proposed partial-episode bootstrapping (PEB), White (2017) introduces a way to consider episodic tasks as a continuing one with a variable discount factor be-tween them.”
Hyperbolic Discounting and Learning over Multiple Horizons (2019, background): “RL researchers have recently adopted more flexible versions beyond a fixed discount factor (Feinberg & Shwartz, 1994; Sutton, 1995; Sutton et al., 2011; White, 2017).”