Learning Without State-Estimation in Partially Observable Markovian Decision Processes

By Satinder Singh, Tommi S. Jaakkola, Michael I. Jordan. 1994

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Memory-based control with recurrent neural networks (2015, background): “As discussed in the literature [25, 20] it is possible to construct examples where deterministic policies perform poorly under partial observability.”
Infinite-Horizon Policy-Gradient Estimation (2001, background): “In fact, the expected discounted reward is simply times the average reward ( Singh et al., 1994, Fact 7), so the gradient of the expected discounted reward is proportional to the gradient of the average reward.”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “One may consider previously proposed semi-batch RL algorithms (Ernst et al., 2005; Lange et al., 2012; Singh et al., 1994; Roux, 2016) or theoretical analysis of switching cost under the tabular PAC-MDP settings (Bai et al., 2019; Guo & Brunskill, 2015) as approaching this issue.”
Policy Tree: Adaptive Representation for Policy Gradient (2015, background): “Value function based algorithms are not guaranteed to work well in the presence of partial observability (Singh, Jaakkola, and Jordan 1994) or when function approximation is used (Boyan and Moore 1995), and so learning an accurate representation of state is important.”
Exponential Family Predictive Representations of State (2007, background): “It is known that the optimal reactive policy may be stochastic (whereas theoptimal policy for POMDPs and MDPs is always deterministic), and it is known that thebest memoryless policy can be arbitrarily suboptimal in the worst case (Singh et al., 1994).”
Policy Gradients Incorporating the Future (2021, methodology): “When applied to such non-Markovian domains, model-free algorithms relying on bootstrapping and value function approximation tend to be biased [37].”
Policy Gradient Critics (2007, methodology): “If the amount of state uncertainty reaches undesirable levels, it may be appropriate not to use TDlearning techniques to train the Policy Gradient Critic, since conventional TD-updates are essentially flawed in hidden state situations with discounted payoff [2].”
On Q-learning Convergence for Non-Markov Decision Processes (2018, methodology): “[Singh et al. , 1994] investigate a direct application of model-free algorithms to POMDPs without a state estimation step akin to our setup but limited to ﬁnite state POMDPs only.”
Reinforcement Learning in POMDP's via Direct Gradient Ascent (2000, unknown): “A curious fact about the present setting is that optimizing the long-term average reward is the same as optimizing expected discounted reward, since η α ( θ ) = η ( θ ) / (1 − α ) (Singh et al., 1994, Fact 7).”
Approximate policy iteration: a survey and some new methods (2011, methodology): “It was introduced in the simulation-based approximate DP context, mostly in the form of value iteration; see Singh et al. [ 43 , 44], Gordon [45], Tsitsiklis and Van Roy [46], and Van Roy [47].”