Neural Predictive Belief Representations

By Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, B. '. Pires, Tobias Pohlen, R´emi Munos. 2018

82 citations recorded by the source corpus.

Unsupervised representation learning has succeeded with excellent results in many applications. It is an especially powerful tool to learn a good representation of environments with partial or noisy observations. In partially observable domains it is important for the representation to encode a belief state, a sufficient statistic of the observations seen so far. In this paper, we investigate whether it is possible to learn such a belief representation using modern neural architectures. Specifically, we focus on one-step frame prediction and two variants of contrastive predictive coding (CPC) as the objective functions to learn the representations. To evaluate these learned representations, we test how well they can predict various pieces of information about the underlying state of the environment, e.g., position of the agent in a 3D maze. We show that all three methods are able to learn belief representations of the environment, they encode not only the state information, but also its uncertainty, a crucial aspect of belief states. We also find that for CPC multi-step predictions and action-conditioning are critical for accurate belief representations in visually complex environments. The ability of neural representations to capture the belief information has the potential to spur new advances for learning and planning in partially observable domains, where leveraging uncertainty is essential for optimal decision making.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “, 1998), latent embedding of (belief) states (Guo et al., 2018) and goals (Nachum et al.”
Causally Correct Partial Models for Reinforcement Learning (2020, methodology): “These challenges have motivated researchers to consider simpler models, henceforth referred to as partial models , i.e. models which are neither conditioned on, nor generate the full set of observed data (Oh et al., 2017; Amos et al., 2018; Guo et al., 2018; Gregor et al., 2019).”
Shaping Belief States with Generative Environment Models for RL (2019, methodology): “The second component is the unsupervised model, which can be: (i) a contrastive loss based on action-conditional CPC [20]; (ii) a deterministic predictive model (similar to [11]) and (iii) an expressive generative predictive model based on ConvDRAW, [21].”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “Using the same DD-PPO baseline architecture from the previous work, this work adds three auxiliary tasks: action-conditional contrastive predictive coding (CPC—A) [96], inverse dynamics [46] and temporal distance estimation.”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “We can also encourage mutual information between model states and observations by instead predicting the states from the images (Guo et al., 2018).”
Pushing it out of the Way: Interactive Visual Navigation (2021, methodology): “During the training stage, the CPC|A utilizes Contrastive Predictive Coding as an auxiliary loss to perform predictive representation learning.”
Data-Efficient Reinforcement Learning with Self-Predictive Representations (2020, unknown): “CPC (Oord et al., 2018), CPC|Action (Guo et al., 2018), ST-DIM (Anand et al., 2019) and DRIML (Mazoure et al., 2020) propose to optimize various temporal contrastive losses in reinforcement learning environments.”
Generalization of Reinforcement Learners with Working and Episodic Memory (2019, background): “Recently it has been shown that agents augmented with one-step contrastive predictive coding (van den Oord et al., 2018, CPC) can learn belief state representations of the environment (Guo et al., 2018).”
Offline Meta Reinforcement Learning - Identifiability Challenges and Effective Data Collection Strategies (2021, methodology): “While other neural belief representations could be used [15], we chose to focus on VariBAD for concreteness.”
Meta reinforcement learning as task inference (2019, background): “Unsupervised learning of the belief state as a means of facilitating learning in POMDPs has been proposed in (Guo et al., 2019; Moreno et al., 2018).”
Learning Causal State Representations of Partially Observable Environments (2019, methodology): “[13] also use neural methods to learn belief states with next-step prediction [17, 18] learn PSRs with RNNs and spectral methods and use policy gradient with an alternating optimization method for the policy and the state representation to handle continuous action spaces.”
Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective (2019, background): “ewards lack an observation tampering incentive. For example, in the rocks and diamonds environment with partial observations, the agent need to maintain a belief state about the environment dynamics (Guo et al., 2018), so that it can navigate to the goal area even after its observations have been ta”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, background): “However, prior work has found it challenging to uphold this promise [7, 79, 118, 125], prompting many prior methods to employ separate objectives for representation learning and RL [31, 43, 70, 74, 89, 91, 100, 101, 117, 140, 143].”
Decoupling Representation Learning from Reinforcement Learning (2020, background): “, 2018) applied contrastive losses over multiple time steps as an auxiliary task for the convolutional and recurrent layers of RL agents, and it has been extended with future action-conditioning (Guo et al., 2018).”
Auxiliary Tasks Speed Up Learning PointGoal Navigation (2020, unknown): “Self-supervised auxiliary tasks, such as next-step visual feature prediction [18], predictive model-2 ing [17, 19], or spatio-temporal mutual information maximization [20, 23] derive supervision from the agent’s own experience.”
Action and Perception as Divergence Minimization (2020, background): “In latent space, it leads to contrastive learning as in NCE (Gutmann and Hyvärinen, 2010), CPC (Oord et al., 2018; Guo et al., 2018), CEB (Fischer, 2020), and SimCLR (Chen et al.”
How to Train PointGoal Navigation Agents on a (Sample and Compute) Budget (2020, unknown): “Sample efﬁciency has been studied extensively in model-free RL, with works proposing new training meth-ods [32, 33], auxiliary losses [34, 35, 36], intrinsic re-wards [37], pre-training [25], and data-augmentation [38, 39] to improve sample efﬁciency.”
Contrastive Representation Learning: A Framework and Review (2020, background): “In addition to learning representations of observations in the environment, CPC|Action [36] is a variant of CPC that explored whether contrastive learning methods can also encode belief states (i.”
World Discovery Models (2019, unknown): “For a ﬁxed t ≥ 0 , the loss used to trained f τ is a cross entropy loss (For a more detailed description of the approach see Guo et al., 2018): .”
Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning (2020, methodology): “…better captures the shared, latent structure across tasks, with the potential to generalize to new unseen tasks Interesting future work also includes a more in-depth study of what predictive representations learn to encode, perhaps using a glass-box approach (Guo et al., 2018; Amos et al., 2018).”