Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization
By P. Liotet, Francesco Vidaich, Alberto Maria Metelli, Marcello Restelli. 2021
11 citations recorded by the source corpus.
Learning in a lifelong setting, where the dynamics continually evolve, is a hard challenge for current reinforcement learning algorithms. Yet this would be a much needed feature for practical applications. In this paper, we propose an approach which learns a hyper-policy, whose input is time, that outputs the parameters of the policy to be queried at that time. This hyper-policy is trained to maximize the estimated future performance, efficiently reusing past data by means of importance sampling, at the cost of introducing a controlled bias. We combine the future performance estimate with the past performance to mitigate catastrophic forgetting. To avoid overfitting the collected data, we derive a differentiable variance bound that we embed as a penalization term. Finally, we empirically validate our approach, in comparison with state-of-the-art algorithms, on realistic environments, including water resource management and trading.
Subfields: Artificial Intelligence, Management Science and Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Angela Bitto-Nemling
- Hamid Eghbalzadeh
- Fabian Paischer
- Vihang Patil
- Sepp Hochreiter
- J. Urgen Schmidhuber
- Quoc V. Le
- Kevin Gimpel
In Management Science and Operations Research
Citations in the corpus
- Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learning (2022, background): “Prior work has highlighted the importance of correctly re-using collected experiences in off-policy algorithms when non-stationarities are encountered (Liotet et al., 2021; Chandak et al., 2020; Xie et al., 2021).”