Spatial Action Maps for Mobile Manipulation
By Jimmy Wu, Xingyuan Sun, Andy Zeng, Shuran Song, Johnny Lee, Szymon Rusinkiewicz, Thomas A. Funkhouser. 2020
88 citations recorded by the source corpus.
Typical end-to-end formulations for learning robotic navigation involve predicting a small set of steering command actions (e.g., step forward, turn left, turn right, etc.) from images of the current state (e.g., a bird's-eye view of a SLAM reconstruction). Instead, we show that it can be advantageous to learn with dense action representations defined in the same domain as the state. In this work, we present "spatial action maps," in which the set of possible actions is represented by a pixel map (aligned with the input image of the current state), where each pixel represents a local navigational endpoint at the corresponding scene location. Using ConvNets to infer spatial action maps from state images, action predictions are thereby spatially anchored on local visual features in the scene, enabling significantly faster learning of complex behaviors for mobile manipulation tasks with reinforcement learning. In our experiments, we task a robot with pushing objects to a goal location, and find that policies learned with spatial action maps achieve much better performance than traditional alternatives.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Silvio Savarese
- Li Fei-Fei
- Roberto Mart'in-Mart'in
- Alexander Toshev
- Ajay Mandlekar
- Danfei Xu
- Fei Xia
- Sergey Levine
In Computer Vision and Pattern Recognition
- Silvio Savarese
- Jitendra Malik
- Luc Van Gool
- Li Fei-Fei
- Xiaogang Wang
- Trevor Darrell
- Dacheng Tao
- Andrea Vedaldi
In Control and Systems Engineering
- Silvio Savarese
- Li Fei-Fei
- Ajay Mandlekar
- Roberto Mart'in-Mart'in
- Danfei Xu
- Chengshu Li
- Fei Xia
- Alexander Toshev
Citations in the corpus
- Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control (2021, methodology): “Our idea of spatially-invariant action space is inspired by a series of work by Zeng et al. and colleagues [34, 37, 39–42].”
- ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation (2020, methodology): “[33] is the most similar method to ours.”
- Diffusion policy: Visuomotor policy learning via action diffusion (2023, unknown): “A popular approach to model multimodal action distributions while maintaining the simplicity of direction action mapping is convert the regression task into classi fi cation by discretizing the action space (Zeng et al., 2021; Wu et al., 2020; Avigal et al., 2022).”
- Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks (2020, unknown): “This technique [67] has shown promising results for robotic warehouse order picking [71], pushing and grasping [70], tossing objects [69], kit assembly [66], and mobile manipulation [62].”
- ReLMoGen: Integrating Motion Generation in Reinforcement Learning for Mobile Manipulation (2021, methodology): “This parametrization is similar to the “spatial action maps” by Wu et al. [34].”