Addressing Function Approximation Error in Actor-Critic Methods

By Scott Fujimoto, Herke Van Hoof, David Meger. 2018

7,119 citations recorded by the source corpus.

In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.

Subfields: Artificial Intelligence, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Pieter Abbeel
  3. Yoshua Bengio
  4. Chelsea Finn
  5. Ruslan Salakhutdinov
  6. Mohit Bansal
  7. Quoc V. Le
  8. Trevor Darrell

In Reinforcement Learning

  1. Sergey Levine
  2. Pieter Abbeel
  3. Chelsea Finn
  4. Ofir Nachum
  5. Shixiang Shane Gu
  6. Aviral Kumar
  7. George Tucker
  8. Shimon Whiteson

In Optimization & Training

  1. Sergey Levine
  2. Yoshua Bengio
  3. Quoc V. Le
  4. Chelsea Finn
  5. Pieter Abbeel
  6. Razvan Pascanu
  7. Kyunghyun Cho
  8. Frank Hutter

Citations in the corpus