Reinforcement Learning without Ground-Truth State

By Xingyu Lin, Harjatin Singh Baweja, David Held. 2019

27 citations recorded by the source corpus.

To perform robot manipulation tasks, a low-dimensional state of the environment typically needs to be estimated. However, designing a state estimator can sometimes be difficult, especially in environments with deformable objects. An alternative is to learn an end-to-end policy that maps directly from high-dimensional sensor inputs to actions. However, if this policy is trained with reinforcement learning, then without a state estimator, it is hard to specify a reward function based on high-dimensional observations. To meet this challenge, we propose a simple indicator reward function for goal-conditioned reinforcement learning: we only give a positive reward when the robot's observation exactly matches a target goal observation. We show that by relabeling the original goal with the achieved goal to obtain positive rewards (Andrychowicz et al., 2017), we can learn with the indicator reward function even in continuous state spaces. We propose two methods to further speed up convergence with indicator rewards: reward balancing and reward filtering. We show comparable performance between our method and an oracle which uses the ground-truth state for computing rewards. We show that our method can perform complex tasks in continuous state spaces such as rope manipulation from RGB-D images, without knowledge of the ground-truth state.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation (2020, neutral): “We refer to [44, 45, 46, 47] for recent works towards this challenge.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “Baselines: HER [80] is a prototypical actor-critic method that uses hindsight relabeling [6]; Goal-conditioned behavioral cloning (GCBC) [22, 41, 83, 116] performs behavior cloning on relabeled experience; model-based fits a density model to the discounted state occupancy measure, similar on [21, 23”
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills (2021, methodology): “Q-learning with hindsight relabeling , analogously to hindsight experience replay (HER) (Andrychowicz et al., 2017) and other prior work (Lin et al., 2019; Fujita et al., 2020).”