Conservative and Greedy Approaches to Classification-Based Policy Iteration

By Mohammad Ghavamzadeh, Alessandro Lazaric. 2012

10 citations recorded by the source corpus.

The existing classification-based policy iteration (CBPI) algorithms can be divided into two categories: direct policy iteration (DPI) methods that directly assign the output of the classifier (the approximate greedy policy w.r.t.~the current policy) to the next policy, and conservative policy iteration (CPI) methods in which the new policy is a mixture distribution of the current policy and the output of the classifier. The conservative policy update gives CPI a desirable feature, namely the guarantee that the policies generated by this algorithm improve at each iteration. We provide a detailed algorithmic and theoretical comparison of these two classes of CBPI algorithms. Our results reveal that in order to achieve the same level of accuracy, CPI requires more iterations, and thus, more samples than the DPI algorithm. Furthermore, CPI may converge to suboptimal policies whose performance is not better than DPI's.

Subfields: Artificial Intelligence, Computer Networks and Communications, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Electrical and Electronic Engineering

Citations in the corpus

Approximate Policy Iteration Schemes: A Comparison (2014, methodology): “A comparison of API and CPI based on the two known bounds was done by Ghavamzadeh & Lazaric (2012).”
Robust Policy Optimization with Baseline Guarantees (2015, background): “Similar approaches in the model-free setup can be found in [7, 10, 20], where safe policy evaluation takes place during exploration.”