Policy Transfer using Reward Shaping

By Tim Brys, Ani Harutyunyan, Matthew E. Taylor, Ann Nowé. 2015

91 citations recorded by the source corpus.

Transfer learning has proven to be a wildly successful approach for speeding up reinforcement learning. Techniques often use low-level information obtained in the source task to achieve successful transfer in the target task. Yet, a most general transfer approach can only assume access to the output of the learning algorithm in the source task, i.e. the learned policy, enabling transfer irrespective of the learning algorithm used in the source task. We advance the state-of-the-art by using a reward shaping approach to policy transfer. One of the advantages in following such an approach, is that it firmly grounds policy transfer in an actively developing body of theoretical research on reward shaping. Experiments in Mountain Car, Cart Pole and Mario demonstrate the practical usefulness of the approach.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Disentangling Transfer in Continual Reinforcement Learning (2022, neutral): “[30, 5] apply reward function reshaping.”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, methodology): “One work by [77] proposed to use RS to transfer an expert policy from the source domain M s to the target domain M t .”
Reward Shaping with Recurrent Neural Networks for Speeding up On-Line Policy Learning in Spoken Dialogue Systems (2015, background): “However our supposition is that RNNs could be trained for reward prediction which are substantially domain independent and hence have wider applications via domain adaptation and extension (Gašić et al., 2015; Brys et al., 2015).”
The rise and potential of large language model based agents: a survey (2025, unknown): “Consequently, researchers introduce transfer learning to expedite an agent’s learning on new tasks [57–59].”
Imitation Learning (2017, background): “In Brys et al. [2015b], transfer learning is applied from a simple version of the problem to a more complex one (e.g., 2D to 3D mountain car and a Mario game without enemies to a game with enemies).”
Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search (2020, background): “For instance, reward shaping [50, 79, 16, 7, 74] leverages the expertise knowledge to reshape the reward distributions to steer the agent’s action selection and navigation towards the expected trajectories; [59, 84, 39, 31, 49, 38, 74] make use of external demonstrations from different sources such…”
Boosted Curriculum Reinforcement Learning (2022, background): “PPR requires the user to choose a decay factor ν to anneal the likelihood of executing the source task policy via p ( π source ) = ν t , where t is the number of past interactions (note that we follow the interpretation of PPR by Brys et al. (2015) for transfer from a single source task).”