InfoBot: Transfer and Exploration via the Information Bottleneck

By Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Sergey Levine, Yoshua Bengio. 2019

189 citations recorded by the source corpus.

A central challenge in reinforcement learning is discovering effective policies for tasks where rewards are sparsely distributed. We postulate that in the absence of useful reward signals, an effective exploration strategy should seek out {\it decision states}. These states lie at critical junctions in the state space from where the agent can transition to new, potentially unexplored regions. We propose to learn about decision states from prior experience. By training a goal-conditioned policy with an information bottleneck, we can identify decision states by examining where the model actually leverages the goal state. We find that this simple mechanism effectively identifies decision states, even in partially observed settings. In effect, the model learns the sensory cues that correlate with potential subgoals. In new environments, this model can then identify novel subgoals for further exploration, guiding the agent through a sequence of potential decision states and through new regions of the state space.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Statistical and Nonlinear Physics

Citations in the corpus

CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, background): “…has been historically studied from the perspective of probabilistic and causal reasoning (Pearl, 2009), cognitive psychology (Grifﬁths & Tenenbaum, 2005), and more recently in the context of machine learning (Goyal et al., 2019b; Sch ¨ olkopf, 2019; Baradel et al., 2019; Bakhtin et al., 2019).”
Recurrent Independent Mechanisms (2019, background): “State dependent activation of different primitive policies was also studied in (Goyal et al., 2019b), and the authors showed that they can learn different primitives, but they also consider that only a single primitive can be active at a particular time step.”
Learning Nearly Decomposable Value Functions Via Communication Minimization (2019, background): “Social inﬂuence (Jaques et al., 2019) and InfoBot (Goyal et al., 2019) penalize message that has no effect on policies of other agents.”
Multitask Soft Option Learning (2019, background): “On the other hand, Goyal et al. (2019) also learn a hierarchical agent, but not to transfer skills, but to find decisions states based on how much information is encoded in the latent layer.”
Toward Robust Long Range Policy Transfer (2021, methodology): “Some methods intend to limit dependency between pretraining policy and task-specific information (Goyal et al. 2019; Galashov et al. 2019) by using information bottleneck (Alemi et al.”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, methodology): “Goyal et al. (2019) consider identifying bottleneck states based on objective similar to eq.”
Predictive Information Accelerates Learning in RL (2020, background): “It is well-known that compressed representations can generalize better in many machine learning settings [31, 3, 7], including RL [18, 10, 9].”
Revisiting Training Strategies and Generalization Performance in Deep Metric Learning (2020, background): “…models has been well studied (Jiang* et al., 2020; Belghazi et al., 2018; Goyal et al., 2017), e.g. in the light of compression (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017) which is covered by strong experimental support (Goyal et al., 2019; Belghazi et al., 2018; Alemi et al., 2016).”
Information asymmetry in KL-regularized RL (2019, unknown): “In concurrent work Goyal et al. (2019) propose an information bottleneck architecture for policies with latent variables that leads to a KL-regularized formulation similar to the one described in Appendix A.2.”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, background): “Another popular approach to learning useful representation is to optimize for effective transfer of the lower-level skills and the learned state abstraction (Gupta et al., 2017; Konidaris & Barto; Goyal et al., 2019).”
Neural Production Systems (2021, methodology): “This is the primary motivation behind NPS.”
What Can Learned Intrinsic Rewards Capture? (2019, background): “, 2017), surprise (Itti & Baldi, 2006), deviation from a default policy (Goyal et al., 2018), weight change (Linke et al.”