f-Divergence constrained policy improvement

By Boris Belousov, Jan Peters. 2017

30 citations recorded by the source corpus.

To ensure stability of learning, state-of-the-art generalized policy iteration algorithms augment the policy improvement step with a trust region constraint bounding the information loss. The size of the trust region is commonly determined by the Kullback-Leibler (KL) divergence, which not only captures the notion of distance well but also yields closed-form solutions. In this paper, we consider a more general class of f-divergences and derive the corresponding policy update rules. The generic solution is expressed through the derivative of the convex conjugate function to f and includes the KL solution as a special case. Within the class of f-divergences, we further focus on a one-parameter family of $\alpha$-divergences to study effects of the choice of divergence on policy improvement. Previously known as well as new policy updates emerge for different values of $\alpha$. We show that every type of policy update comes with a compatible policy evaluation resulting from the chosen f-divergence. Interestingly, the mean-squared Bellman error minimization is closely related to policy evaluation with the Pearson $\chi^2$-divergence penalty, while the KL divergence results in the soft-max policy update and a log-sum-exp critic. We carry out asymptotic analysis of the solutions for different values of $\alpha$ and demonstrate the effects of using different divergence functions on a multi-armed bandit problem and on common standard reinforcement learning problems.

Subfields: Political Science and International Relations

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Political Science and International Relations

1. Bo Dai
2. Ofir Nachum

Citations in the corpus

• Reinforcement Learning via Fenchel-Rockafellar Duality (2020, neutral): “This form of value learning (and K learning) before recovering a policy from the optimal values is the same as derived by Belousov and Peters (2017).”
• AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, methodology): “An approach more similar to ours is given by Belousov and Peters (2017), which regularizes the max-return objective using an f -divergence over state-action densities.”
• Imitation Learning as f-Divergence Minimization (2019, background): “[50] studied how choice of divergence affected policy improvement for reinforcement learning.”