Reinforcement Learning for Improving Agent Design

By David R Ha. 2018

154 citations recorded by the source corpus.

In many reinforcement learning tasks, the goal is to learn a policy to manipulate an agent, whose design is fixed, to maximize some notion of cumulative reward. The design of the agent's physical structure is rarely optimized for the task at hand. In this work, we explore the possibility of learning a version of the agent's design that is better suited for its task, jointly with the policy. We propose an alteration to the popular OpenAI Gym framework, where we parameterize parts of an environment, and allow an agent to jointly learn to modify these environment parameters along with its policy. We demonstrate that an agent can learn a better structure of its body that is not only better suited for the task, but also facilitates policy learning. Joint learning of policy and structure may even uncover design principles that are useful for assisted-design applications.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jeff Clune
2. Sergey Levine
3. Kenneth O. Stanley
4. Joel Lehman
5. Yoshua Bengio
6. Quoc V. Le
7. Li Fei-Fei
8. Rui Wang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Li Fei-Fei
3. Dacheng Tao
4. Trevor Darrell
5. Xiaogang Wang
6. Andrea Vedaldi
7. Jitendra Malik
8. Ming-Hsuan Yang

In Control and Systems Engineering

1. Kenneth O. Stanley
2. Joel Lehman
3. Jeff Clune
4. Rui Wang
5. Silvio Savarese
6. Adam Gaier
7. Li Fei-Fei
8. C. Freeman

Citations in the corpus

• Weight Agnostic Neural Networks (2019, methodology): “The WANN architecture not only solves the task without training the individual weights, but uses only 210 connections, an order of magnitude fewer than commonly used topologies (2804 connections used in the SOTA baseline [37]).”
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions (2019, methodology): “Following the architecture of controllers for the same domain by Ha [56], all controllers in the experiments are implemented as neural networks with 3 fully-connected layers with tanh activation functions.”
• Neuroevolution of self-interpretable agents (2020, unknown): “Neuroevolution-based methods for tackling challenging RL tasks have recently gained popularity due to their simplicity and competitiveness to Deep RL methods, even on vision-based RL benchmark tasks [34, 36, 62, 81, 94].”
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence (2019, background): “Such specialization is especially easy to imagine if the bodies of virtual robots are simultaneously optimized, which is a subfield with a long history [158, 21, 22, 67, 6, 57].”
• POET: open-ended coevolution of environments and their optimized solutions (2019, methodology): “Gym [30] used in Ha [21] (denoted as reference in Table 1).”
• Learning to Predict Without Looking Ahead: World Models Without Forward Prediction (2019, methodology): “While it’s possible to optimize this objective with any reinforcement learning method [37, 38, 52, 53], we choose to use population based REINFORCE [67] due to its simplicity and effectiveness at achieving high scores on various tasks [18, 19, 49].”
• Embodied intelligence via learning and evolution (2021, background): “Hence, prior work has either evolved agents in severely limited morphological search spaces 9–13 or focused on finding optimal parameters given a fixed hand designed morphology 14–16 .”