Compress and Control

By Joel Veness, Marc G. Bellemare, Marcus Hutter, Alvin Chua, Guillaume Desjardins. 2014

30 citations recorded by the source corpus.

This paper describes a new information-theoretic policy evaluation technique for reinforcement learning. This technique converts any compression or density model into a corresponding estimate of value. Under appropriate stationarity and ergodicity conditions, we show that the use of a sufficiently powerful model gives rise to a consistent value function estimator. We also study the behavior of this technique when applied to various Atari 2600 video games, where the use of suboptimal modeling techniques is unavoidable. We consider three fundamentally different models, all too limited to perfectly model the dynamics of the system. Remarkably, we find that our technique provides sufficiently accurate value estimates for effective on-policy control. We conclude with a suggestive study highlighting the potential of our technique to scale to large problems.

Subfields: Artificial Intelligence, Management Science and Operations Research, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Efficiency & Compression

Citations in the corpus

Shaking the foundations: delusions in sequence models for interaction and control (2021, background): “This is a simple question with far-reaching implications on foundation models [4], language agents [5], behavior cloning [10], inverse reinforcement learning [1, 13], goal-/return-conditioned policies [6, 11, 22]), and so forth.”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, background): “Although an error-free model might be unattainable, there is plenty of evidence that even coarse value functions are sufficient for the model-free case (Veness et al., 2015), raising the question of how to compensate for a model’s flaws.”
Unifying Count-Based Exploration and Intrinsic Motivation (2016, methodology): “When Y is small and ﬁnite and we are given a sequential density model ρ over X , we can construct a reasonable sequential density model over the joint alphabet X × Y by means of a chain rule-like construct (Veness et al., 2015).”
Increasing the Action Gap: New Operators for Reinforcement Learning (2015, background): “…to extend the consistent Bellman operator to Q-value approximation schemes which lack a probabilistic interpretation, such as linear approximation (Sutton 1996), locally weighted regression (Atkeson 1991), neural networks (Tesauro 1995), or even information-theoretic methods (Veness et al. 2015).”
A Distributional Perspective on Reinforcement Learning (2017, background): “Related to this dual is the Compress and Control algorithm Veness et al. (2015), which describes a value function by learning a return distribution using density models.”
Online Learning with Gated Linear Networks (2017, methodology): “…require multiple passes over the dataset to work well, which makes it signiﬁcantly more challenging to incorporate them into naturally online settings such as reinforcement learning, where density modeling is just starting to see promising applications (Veness et al., 2015; Ostrovski et al., 2017).”
A Deep Learning Approach for Joint Video Frame and Reward Prediction in Atari Games (2016, methodology): “In recent work, model-based techniques were successfully used to learn statistics about cumulative future rewards (Veness et al., 2015) and to improve exploration (Pathak et al., 2017; Bellemare et al., 2016; Oh et al., 2015), resulting in more data efﬁcient learning compared to model-free…”
Counterfactual Credit Assignment in Model-Free Reinforcement Learning (2020, background): “A first approach, related to distributional reinforcement learning (Veness et al., 2015; Bellemare et al., 2017), is to forego modeling the environment (as in section E.”
State of the Art Control of Atari Games Using Shallow Reinforcement Learning (2015, methodology): “[30], for example, propose the use of compression techniques as a policy evaluation approach in RL, evaluating their method in the ALE.”
The Forget-me-not Process (2016, methodology): “Alternatively, the FMN process could be used to augment the conditional state density models used for value estimation in [18].”