Sim-to-Real Reinforcement Learning for Deformable Object Manipulation

By Jan Matas, Stephen James, Andrew J. Davison. 2018

424 citations recorded by the source corpus.

We have seen much recent progress in rigid object manipulation, but interaction with deformable objects has notably lagged behind. Due to the large configuration space of deformable objects, solutions using traditional modelling approaches require significant engineering work. Perhaps then, bypassing the need for explicit modelling and instead learning the control in an end-to-end manner serves as a better approach? Despite the growing interest in the use of end-to-end robot learning approaches, only a small amount of work has focused on their applicability to deformable object manipulation. Moreover, due to the large amount of data needed to learn these end-to-end solutions, an emerging trend is to learn control policies in simulation and then transfer them over to the real world. To-date, no work has explored whether it is possible to learn and transfer deformable object policies. We believe that if sim-to-real methods are to be employed further, then it should be possible to learn to interact with a wide variety of objects, and not only rigid objects. In this work, we use a combination of state-of-the-art deep reinforcement learning algorithms to solve the problem of manipulating deformable objects (specifically cloth). We evaluate our approach on three tasks --- folding a towel up to a mark, folding a face towel diagonally, and draping a piece of cloth over a hanger. Our agents are fully trained in simulation with domain randomisation, and then successfully deployed in the real world without having seen any real deformable objects.

Subfields: Artificial Intelligence, Computational Mechanics, Control and Systems Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Control and Systems Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, methodology): “Such approaches are termed Sim2Real [Golemo, 2018], and have been successfully applied [Christiano et al., 2016, Matas et al., 2018] in many scenarios.”
SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation (2020, background): “Some recent works have used learning-based methods to explore the challenges of deformable object manipulation [9, 10, 11, 12]; however, these works often each evaluate on a different task variant with different simulators or robot setups, making it challenging to directly compare these approaches.”
RLBench: The Robot Learning Benchmark & Learning Environment (2019, background): “…on-the-ﬂy, meaning that you cam generate an inﬁnite amount for your imitation learning algorithm. c) Sim-to-Real Transfer : Recently there has been a large amount of work in learning control policies in simulation and then transferring these to the real world [48], [49], [50], [51], [52], [53].”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, background): “ids [35], door opening [44, 3], and cloth folding [28].”
Task-Embedded Control Networks for Few-Shot Imitation Learning (2018, methodology): “To enable transfer, we use domain randomisation; a method that is increasingly being used transfer learned polices from simulation to the real world [3, 30, 31].”
Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks (2018, methodology): “ndomization during training (i.e. is not destabilized). This is contrary to other RL methods, such as DDPG [35] and A3C [39], where heavy domain randomization has been shown to cause training to fail [38, 69]. Although QT-Opt was able to train stably with randomization, the results show that this do”
A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learning (2018, background): “Recent work [8, 16] has shown that off-policy RL can be combined with human demonstrations to allow neuralnetwork controllers to be trained efficiently, even when using a sparse reward function (e.”
Time Reversal as Self-Supervision (2018, background): “namics are reversible and TRASS can be applied, for example turning a knob or closing a drawer. Irreversible Object Deformations: In cases where object deformations can be reversed like cloth folding [39] or rope manipulation [40] TRASS is applicable. However in cases where object deformations canno”
PyRep: Bringing V-REP to Deep Robot Learning (2019, background): “• Training on a large amount of simulation data with potentially a small amount of real-world data, and ﬁnd ways of transferring this knowledge from simulation to the real world [4, 5, 6, 7, 8].”
COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning (2020, background): “Robotic RL. RL has been applied to a wide variety of robotic manipulation tasks, including grasping objects [11, 12], in-hand object manipulation [13, 14, 15, 16], pouring ﬂuids [17], door opening [18, 19], and manipulating cloth [20, 21].”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, positive): “Domainrandomization[265], [266],[267]isalsoasimpleandeffectivesim-to-realtechnique forRLpolicylearning,whichisfurtheradaptedforend-to-end autonomousdriving[188],[268].”
Deep Imitation Learning of Sequential Fabric Smoothing Policies (2019, methodology): “To prevent potential damage to the grippers, the fabric plane is foam rubber, which allows us to liberally set the gripper height to be lower and avoids a source of height error present in [26].”
“Good Robot!”: Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer (2019, background): “Some of the most closely related recent work involves tasks with multiple actions: [36] includes placing one block on another, [37] places one towel on a bar, and [38] clears a bin, but the ﬁrst two are not long-horizon tasks and the possibility of progress reversal (Fig.”
Scaling data-driven robotics with reward sketching and batch reinforcement learning (2019, methodology): “Even with careful simulation design there remains a domain gap between simulation and reality that needs to be dealt with [67, 48].”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, neutral): “Domain Randomization [227], [228], [229] increases model generalization by varying simulation parameters to cover real-world conditions.”
RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer (2020, methodology): “With domain randomization, a policy is trained with randomized simulation parameters and scene configurations which produce differences in visual appearance [6], [7], [8], [9], [10], [11].”
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation (2019, background): “…from pixels (Ghadirzadeh et al., 2017; Levine et al., 2015; Kalashnikov et al., 2018; Lange et al., 2012; OpenAI et al., 2018; Schenck & Fox, 2016; Matas et al., 2018; James et al., 2017; Singh et al., 2019), including in goal-conditioned settings (Kaelbling, 1993; Schaul et al., 2015;…”
Learning to Manipulate Deformable Objects without Demonstrations (2019, background): “However, many of these works [19], [45] require expert demonstrations to guide learning for cloth manipulation.”