An Intelligent Battery Controller Using Bias-Corrected Q-learning
By Donghun Lee, Warrren B Powell. 2012
31 citations recorded by the source corpus.
The transition to renewables requires storage to help smooth short-term variations in energy from wind and solar sources, as well as to respond to spikes in electricity spot prices, which can easily exceed 20 times their average. Efficient operation of an energy storage device is a fundamental problem, yet classical algorithms such as $Q$-learning can diverge for millions of iterations, limiting practical applications. We have traced this behavior to the max-operator bias, which is exacerbated by high volatility in the reward function, and high discount factors due to the small time steps. We propose an elegant bias correction procedure and demonstrate its effectiveness.
Subfields: Automotive Engineering, Biomedical Engineering, Electrical and Electronic Engineering, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Automotive Engineering
In Electrical and Electronic Engineering
- Roy Fox
- Arsenii Kuznetsov
- Naftali Tishby
- Warrren B Powell
- Alexander Grishin
- Dmitry P. Vetrov
- Ari Pakman
- Pavel Shvechikov
In Reinforcement Learning
- Roy Fox
- Ari Pakman
- Naftali Tishby
- Arsenii Kuznetsov
- Warrren B Powell
- Alexander Grishin
- Donghun Lee
- Dmitry P. Vetrov
Citations in the corpus
- Taming the Noise in Reinforcement Learning via Soft Updates (2015, background): “Other modifications of Qlearning that attempt to reduce or correct the bias are suggested in [12, 13].”
- Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (2020, background): “The underestimation concept became much more prominent, while the adoption of ”additive compensation” is quite limited to date (Patnaik & Anwar, 2008; Lee & Powell, 2012).”
- G-Learning: Taming the Noise in Reinforcement Learning via Soft Updates (2015, background): “Other modifications of Q-learning that attempt to reduce or correct the bias are suggested in [13, 14].”
- Bias-corrected Q-learning to control max-operator bias in Q-learning (2013, background): “In problems involving energy storage under highly stochastic prices for instance [16], we have encountered practical cases where the initial “divergence” of estimated values of the action-value function (Q-function) still persists after millions of iterations.”