Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards
By Alexander R. Trott, Stephan Zheng, Caiming Xiong, Richard Socher. 2019
142 citations recorded by the source corpus.
While using shaped rewards can be beneficial when solving sparse reward tasks, their successful application often requires careful engineering and is problem specific. For instance, in tasks where the agent must achieve some goal state, simple distance-to-goal reward shaping often fails, as it renders learning vulnerable to local optima. We introduce a simple and effective model-free method to learn from shaped distance-to-goal rewards on tasks where success depends on reaching a goal state. Our method introduces an auxiliary distance-based reward based on pairs of rollouts to encourage diverse exploration. This approach effectively prevents learning dynamics from stabilizing around local optima induced by the naive distance-to-goal reward shaping and enables policies to efficiently solve sparse reward tasks. Our augmented objective does not require any additional reward engineering or domain expertise to implement and converges to the original sparse objective as the agent learns to solve the task. We demonstrate that our method successfully solves a variety of hard-exploration tasks (including maze navigation and 3D construction in a Minecraft environment), where naive distance-based reward shaping otherwise fails, and intrinsic curiosity and reward relabeling strategies exhibit poor performance.
Subfields: Aerospace Engineering, Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Aerospace Engineering
In Artificial Intelligence
- Caiming Xiong
- Richard Socher
- Quoc V. Le
- Mohit Bansal
- Yoshua Bengio
- Jie Tang
- Luke S. Zettlemoyer
- Jianfeng Gao
Citations in the corpus
- RUDDER: Return Decomposition for Delayed Rewards (2018, background): “Sibling Rivalry [48] overcomes local optima of distance-to-goal-based reward shaping by changing the original reward, while still finding an optimal policy for the original reward.”
- Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (2020, methodology): “The maze environments are adapted from the open-source implementation 3 by Trott et al. (2019).”
- ReFT: Reasoning with Reinforced Fine-Tuning (2024, unknown): “For 1 ≤ t ≤ L , we write Such a partial reward can help reduce the effect of learning from sparse reward (Riedmiller et al., 2018; Trott et al., 2019).”
- Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (2022, background): “[54] Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher.”
- Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (2020, methodology): “…incorporate recent advances to our implementation such as exploration with State Marginal Matching (Lee et al., 2019), vector quantization techniques to impose discrete priors in VAEs (van den Oord et al., 2017), and relabeling techniques to optimize deceptive reward functions (Trott et al., 2019).”