Reinforcement Learning Algorithm for Partially Observable Markov Decision Problems

By Tommi S. Jaakkola, Satinder Singh, Michael I. Jordan. 1994

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, background): “However, memories of previous events can help to deal with partially observable Markov decision problems (POMDPs) (e.g., Schmidhuber, 1991c; Ring, 1991, 1993, 1994; Williams, 1992a; Lin, 1993; Teller, 1994; Schmidhuber, 1995; Kaelbling et al., 1995; Littman et al., 1995; Boutilier and Poole, 1996; L”
Shaping Belief States with Generative Environment Models for RL (2019, background): “A key concept in POMDPs is the notion of a belief-state, which can be defined as the sufficient statistics of future states [6, 7, 8].”
Reinforcement Learning in Healthcare: A Survey (2019, background): “the mappings from observations to hidden states [67].”
The Optimal Reward Baseline for Gradient-Based Reinforcement Learning (2001, methodology): “A number of similar algorithms followed, including those in (Glynn, 1986; Cao and Chen, 1997; Cao and Wan, 1998; Fu and Hu, 1994; Jaakkola et al., 1995).”
Multiagent learning using a variable learning rate (2002, background): “There are single-agent learning techniques that are capable of playing stochastic policies [1,2,16,30].”
Infinite-Horizon Policy-Gradient Estimation (2001, background): “…know of, albeit for a slightly different update of the form θ t +1 = θ t + γ t [ r ( X t , s ) − ˆ η ( θ t )] z t , where ˆ η ( θ t ) is a moving estimate of the expected performance, and is also updated on-line (this update was ﬁrst suggested in the context of POMDP s by Jaakkola et al. (1995)).”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “This triggered work on partially observable Markov decision problems (POMDPs) [223, 222, 227, 204, 205, 206, 316, 148, 278, 122, 152, 25, 114, 160, 126, 308, 309, 183].”
Neural Predictive Belief Representations (2018, background): “This is due to the fact that the belief state is a sufficient statistic for predicting future observations and future states, as well as the optimal policy in the RL setting (Rabiner, 1989; Jaakkola et al., 1995).”
Reinforcement Learning with Factored States and Actions (2004, background): “There has been work on reinforcement learning with large state spaces, state uncertainty and partial observability (see for example Bertsekas and Tsitsiklis, 1996; Jaakkola et al., 1995).”
Approximate Gradient Methods in Policy-Space Optimization of Markov Reward Processes (2003, methodology): “Finally, the introduction of a discount factor, mostly with the purpose of limiting the variance of the gradient estimates, appears in Jaakkola et al. (1995) and Kimura et al. (1997), as well as in the more recent references (Baxter and Bartlett, 1999; Cao, 2000).”
The Thing that we Tried Didn't Work very Well: Deictic Representation in Reinforcement Learning (2002, background): “One alternative to value-based learning is direct policy search [17, 7], which is less a ected by problems of partial observability but inherits all the problems that come with local search.”
Incremental self-improvement for life-time multi-agent reinforcement learning (1996, background): “Some reinforcement learning algorithms are guaranteed to converge in certain limited, nonchanging environments, e.g., (Kumar and Varaiya, 1986; Barto, 1989; Watkins and Dayan, 1992; Williams, 1992; Jaakkola et al., 1995; Kaelbling et al., 1995; Ring, 1994).”
Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning (2001, unknown): “Then there are constants C ≤ 5B2R(R + M) and Ω ≤ 4Lτ ln(eT ) such that Var ( 1 T T−1 ∑ t=0 ∇µUt(Yt) µUt(Yt) (Jt+1−Ar(Yt)) ) ≤ Ω T Varπ ( ∇µu(y) µu(y) (Jβ(j)−Ar(y)) ) + Ω T E ( ∇µu(y) µu(y) (J (j) − Jβ(j)) )2 + ( Ω T + 1 ) C (1 − β)2 βT , where, as always, (i, y, u, j) are generated iid with i∼π,…”