Predictive-State Decoders: Encoding the Future into Recurrent Networks

By Arun Venkatraman, Nicholas Rhinehart, Wen Sun, Lerrel Pinto, Martial Hebert, Byron Boots, Kris M. Kitani, J. Bagnell. 2017

46 citations recorded by the source corpus.

Recurrent neural networks (RNNs) are a vital modeling technique that rely on internal states learned indirectly by optimization of a supervised, unsupervised, or reinforcement training loss. RNNs are used to model dynamic processes that are characterized by underlying latent states whose form is often unknown, precluding its analytic representation inside an RNN. In the Predictive-State Representation (PSR) literature, latent state processes are modeled by an internal state representation that directly models the distribution of future observations, and most recent work in this area has relied on explicitly representing and targeting sufficient statistics of this probability distribution. We seek to combine the advantages of RNNs and PSRs by augmenting existing state-of-the-art recurrent neural networks with Predictive-State Decoders (PSDs), which add supervision to the network's internal state representation to target predicting future observations. Predictive-State Decoders are simple to implement and easily incorporated into existing training pipelines via additional loss regularization. We demonstrate the effectiveness of PSDs with experimental results in three different domains: probabilistic filtering, Imitation Learning, and Reinforcement Learning. In each, our method improves statistical performance of state-of-the-art recurrent baselines and does so with fewer iterations and less data.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Recurrent Neural Filters: Learning Independent Bayesian Filtering Steps for Time Series Prediction (2019, methodology): “Predictive state RNNs (PSRNN) (Choromanski et al., 2018; Downey et al., 2017; Venkatraman et al., 2017) use an alternative formulation of the Bayes ﬁlter, utilising a state representation that corresponds to the statistics of a distribution for future observations.”
Learning Awareness Models (2018, background): “Several authors have implemented intrinsic motivation, or curiosity based objectives in visual space, through predicting interactions with objects (Pinto et al., 2016), or through predicting summary statistics of the future (Venkatraman et al., 2017; Downey et al., 2017).”
r2p2: A ReparameteRized Pushforward Policy for Diverse, Precise Generative Path Forecasting (2018, background): “In [10], predictions inform a robot’s behavior, and in [40], policy representations for imitation and reinforcement learning are guided by a future observation forecasting objective.”
Temporal Difference Variational Auto-Encoder (2018, background): “Related to our deﬁnition of belief states as sufﬁcient statistics is the notion of predictive state representations (PSRs) (Littman & Sutton, 2002); see also (Venkatraman et al., 2017) for a model that learns PSRs which, combined with a decoder, can predict future observations.”