Evaluating Real-World Robot Manipulation Policies in Simulation

By Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani and 6 more. Conference on Robot Learning, 2024

457 citations recorded by the source corpus.

The field of robotics has made significant advances towards generalist robot manipulation policies. However, real-world evaluation of such policies is not scalable and faces reproducibility challenges, which are likely to worsen as policies broaden the spectrum of tasks they can perform. We identify control and visual disparities between real and simulated environments as key challenges for reliable simulated evaluation and propose approaches for mitigating these gaps without needing to craft full-fidelity digital twins of real-world environments. We then employ these approaches to create SIMPLER, a collection of simulated environments for manipulation policy evaluation on common real robot setups. Through paired sim-and-real evaluations of manipulation policies, we demonstrate strong correlation between policy performance in SIMPLER environments and in the real world. Additionally, we find that SIMPLER evaluations accurately reflect real-world policy behavior modes such as sensitivity to various distribution shifts. We open-source all SIMPLER environments along with our workflow for creating new environments at https://simpler-env.github.io to facilitate research on general-purpose manipulation policies and simulated evaluation frameworks.

View Paper: https://www.semanticscholar.org/paper/202cc3fcea588cee37ffabcb0b5e145567109a66

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation (2024, unknown): “( b) Scaling behavior: averaged success rate on SIMPLER [33] with respect to action module size.”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, background): “However, when it comes to complex open-world problems, the evaluation procedure must also scale up to be realistic and informative (204).”
UniVLA: Learning to Act Anywhere with Task-centric Latent Actions (2025, unknown): “…the performance of our proposed generalist policy, our evaluation framework assesses the capabilities of UniVLA across a diverse suite of benchmarks (including manipulation benchmarks: LIBERO [48], CALVIN [56], Sim-plerEnv [47], and a navigation benchmark: R2R [4]) and real-world scenarios.”
SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model (2025, unknown): “For the simplerenv [35] evaluation in ??”
Robotic Control via Embodied Chain-of-Thought Reasoning (2024, unknown): “We also tried rolling out the fine-tuned ECoT model in the SIMPLER real-to-sim environments of [72] on the Google Robot tasks, while prompting for ECoT prediction as described above.”