Trading Value and Information in MDPs

By Jonathan Rubin, Ohad Shamir, Naftali Tishby. 2012

104 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Reinforcement Learning, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Reinforcement Learning

In Game & Decision Theory

Citations in the corpus

Taming the Noise in Reinforcement Learning via Soft Updates (2015, methodology): “Information considerations have received attention in recent years in various machine learning settings, with the free energy F π and similar quantities used as a design principle for policies in known MDPs [10, 29, 30].”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “, 2017a), ensuring tractability (Todorov, 2006), and mitigating observation noise (Rubin et al., 2012; Fox et al., 2016).”
Boosting the Actor with Dual Critic (2017, methodology): “Their algorithm is based on the duality of entropy-regularized Bellman equation (Todorov, 2007; Rubin et al., 2012; Fox et al., 2015; Haarnoja et al., 2017; Nachum et al., 2017), rather than the exact Bellman optimality equation used in our work.”
Smoothed Dual Embedding Control (2017, background): “Remark (Connection to TRPO and natural policy gradient): The update of wπ is highly related to trust region policy optimization (TRPO) of Schulman et al. (2015) and natural policy gradient (NPG) (Kakade, 2002; Rajeswaran et al.”
Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning (2015, background): “…problem that forms the basis of a large number of applications in machine learning, including rate distortion theory [4], information bottleneck methods [26], population coding [1], curiosity-driven exploration [24, 19], model selection [3], and intrinsically-motivated reinforcement learning [20].”
An Alternative Softmax Operator for Reinforcement Learning (2016, background): “be derived from information theoretical principles as a way of regularizing policies with a cost function defined by KL divergence (Todorov, 2006; Rubin et al., 2012; Fox et al., 2016).”
SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation (2017, background): “…regularization is introduced in dual form for smoothing (Nesterov, 2005), the entropy-regularized MDP has been proposed for exploration (de Farias & Van Roy, 2000; Haarnoja et al., 2017), taming noise in observations (Rubin et al., 2012; Fox et al., 2016), and ensuring tractability (Todorov, 2006).”
A Separation Principle for Control in the Age of Deep Learning (2017, background): “Similarly [38, 33] deal with capacity costs, but in continuous time.”
Information-Theoretic Bounded Rationality (2015, background): “…al., 2010, Friston, 2010, Peters et al., 2010, Tishby and Polani, 2011, Kappen et al., 2012, Neumann and Peters, 2012, Rawlik and Toussaint, 2012, Rubin et al., 2012, Fox and Tishby, 2012, Neumann and Peters, 2013, Grau-Moya et al., 2013, Tishby and Zaslavsky, 2015, Tanaka et al., 2015, Mohamed…”
Influence-Based Multi-Agent Exploration (2019, unknown): “Mutual information is a critical statistical quantity and has been used in single-agent intrinsically motivated exploration (Rubin et al., 2012; Still & Precup, 2012; Salge et al., 2014; Mohamed & Rezende, 2015; Hyoungseok Kim, 2019) and multi-agent intention sharing and hiding (Strouse et al.,…”
Information asymmetry in KL-regularized RL (2019, methodology): “…ideas can be modeled using tools from information and probability theory (e.g. Tishby & Polani, 2011; Ortega & Braun, 2011; Still & Precup, 2012; Rubin et al., 2012; Ortega & Braun, 2013; Tiomkin & Tishby, 2017), for instance, via constraints on the channel capacity between past states and…”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, background): “Another closely related regularization for RL is using information bottleneck (Tishby & Polani, 2011; Still & Precup, 2012; Rubin et al., 2012; Ortega & Braun, 2013; Tiomkin & Tishby, 2017).”
G-Learning: Taming the Noise in Reinforcement Learning via Soft Updates (2015, methodology): “Information considerations have received attention in recent years in various machine learning settings, with the free energy F and similar quantities used as a design principle for policies in known MDPs [10, 30, 31].”