On Value Functions and the Agent-Environment Boundary
By Nan Jiang. 2019
27 citations recorded by the source corpus.
When function approximation is deployed in reinforcement learning (RL), the same problem may be formulated in different ways, often by treating a pre-processing step as a part of the environment or as part of the agent. As a consequence, fundamental concepts in RL, such as (optimal) value functions, are not uniquely defined as they depend on where we draw this agent-environment boundary, causing problems in theoretical analyses that provide optimality guarantees. We address this issue via a simple and novel boundary-invariant analysis of Fitted Q-Iteration, a representative RL algorithm, where the assumptions and the guarantees are invariant to the choice of boundary. We also discuss closely related issues on state resetting and Monte-Carlo Tree Search, deterministic vs stochastic systems, imitation learning, and the verifiability of theoretical assumptions from data.
Subfields: Artificial Intelligence, Economics and Econometrics, Management Science and Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Masatoshi Uehara
- Nan Jiang
- Jiawei Huang
- Khimya Khetarpal
- Matthew Riemer
- Irina Rish
- Sergey Levine
- Doina Precup
In Economics and Econometrics
In Management Science and Operations Research
- Masatoshi Uehara
- Jiawei Huang
- Doina Precup
- Nan Jiang
- Khimya Khetarpal
- Stuart J. Russell
- Irina Rish
- Banghua Zhu
Citations in the corpus
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Jiang (2019) and Harutyunyan (2020) offer fresh perspectives on this discussion.”
- Minimax Weight and Q-Function Learning for Off-Policy Evaluation (2019, background): “…Bπe (Antos et al., 2008), otherwise ADP can diverge in discounted problems (Tsitsiklis and Van Roy, 1997) or suffer exponential sample complexity in finite-horizon problems (Dann et al., 2018, Theorem 45); we refer the readers to Chen and Jiang (2019) for further discussions on this condition.”
- Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism (2021, background): “, maxπ C [26], [33], [57]–[61] and some definitions are more complex and stronger assuming a bounded ratio per time step [24], [25], [62]–[64].”