Modified Policy Iteration Algorithms for Discounted Markov Decision Problems

By Martin L. Puterman, Moon Chirl Shin. 1978

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

(CAD)$^2$RL: Real Single-Image Flight without a Single Real Image (2016, methodology): “Our method can be interpreted as a modiﬁcation of ﬁtted Q-iteration [24], in the sense that we iteratively reﬁt a Q-function estimator to samples, as well as a variant of modiﬁed policy iteration (MPI) [22], in the sense that we estimate Q-values using multi-step rollouts of the current policy.”
Offline RL Without Off-Policy Evaluation (2021, methodology): “, K do Policy evaluation: Q̂k−1 = E(πk−1, DN , Q̂k−2) Policy improvement: πk = I(Q̂k−1 , β̂,DN , πk−1) end We consider a generic offline approximate modified policy iteration (OAMPI) scheme, shown in Algorithm 1 (and based off of Puterman and Shin [1978], Scherrer et al. [2012]).”
Reinforcement Learning: A Survey (1996, methodology): “Modi(cid:12)ed policy iteration (Puterman & Shin, 1978) seeks a trade-o(cid:11) between cheap and e(cid:11)ective iterations and is preferred by some practictioners (Rust, 1996).”
A unified view of entropy-regularized Markov decision processes (2017, methodology): “Using more value updates leads to a modiﬁed regularized Policy Iteration algorithm (we call it ModRegPI-2 and ModRegPI-20, for 2 and 20 updates, respectively) that interpolates between DPP and TRPO.”
Tight Performance Bounds for Approximate Modified Policy Iteration with Non-Stationary Policies (2013, methodology): “When ` = 1, one recovers the standard MPI algorithm by Puterman and Shin (1978) (that itself generalizes the standard VI and PI algorithm).”
Performance Bounds for Lambda Policy Iteration and Application to the Game of Tetris (2007, methodology): “From this common view of value iteration, it is natural to introduce the well-known modified policy iteration algorithm (Puterman and Shin, 1978) which makes n steps at each update: { πk+1 ← greedy(vk) vk+1 ← (T πk+1)nvk ⇔ { πk+1 ← greedy(vk) vk+1 ← [ I + ...+ (γP πk+1)n−1 ] rπk+1 + (γP πk+1)nvk.”
Exploiting Best-Match Equations for Efficient Reinforcement Learning (2011, methodology): “Using this model, the agent can compute Q (or the value function V ) using dynamic programming methods (Bellman, 1957) such as value iteration (Puterman and Shin, 1978).”
Reinforcement Learning in Continuous State and Action Spaces (2012, methodology): “This can be done iteratively, for instance usingvalue iterationor policy iteration (Bellman, 1957; Howard, 1960; Puterman and Shin, 1978; Puterman, 1994).”
Stochastic dynamic programming with factored representations (2000, background): “work extremely well in practice and can be tuned so that both policy iteration and value iteration are special cases [62, 63 ].”
A Theory of Regularized Markov Decision Processes (2019, methodology): “Here, we consider directly the modiﬁed policy iteration approach [Puterman and Shin, 1978], that we regularize (reg-MPI for short): Given an initial v 0 , reg-MPI iteratively performs a regularized greedy step to get π k +1 and a partial regularized evaluation step to get v k +1 .”
Abstraction and Approximate Decision-Theoretic Planning (1997, background): “Puterman and Shin [46] have observed that the exact value of the current policy 10”
Reinforcement learning design for cancer clinical trials (2009, unknown): “Combining these two methods together, we obtain two other methods called policy iteration and value iteration [15,16].”
Planning, Learning and Coordination in Multiagent Decision Processes (1996, methodology): “While algorithms such as modified policy iteration [43] are often used in practice, an especially simple algorithm is value iteration, based on Bellman’s [4] “principle of optimality.”
Closed-Loop Learning of Visual Control Policies (2011, unknown): “Value Iteration learns the optimal state-action value function Q∗, whereas Policy Iteration and Modified Policy Iteration directly learn an optimal percept-to-action mapping.”
Reinforcement Learning: An Introduction (1998, methodology): “Our discussion of value iteration as a form of truncated policy iteration is based on the approach of Puterman and Shin (1978), who presented a class of algorithms called modified policy iteration, which includes policy iteration and value iteration as special cases.”