Effective Diversity in Population-Based Reinforcement Learning
By Jack Parker-Holder, Aldo Pacchiano, Krzysztof Marcin Choromanski, Stephen J. Roberts. 2020
191 citations recorded by the source corpus.
Exploration is a key problem in reinforcement learning, since agents can only learn from data they acquire in the environment. With that in mind, maintaining a population of agents is an attractive method, as it allows data be collected with a diverse set of behaviors. This behavioral diversity is often boosted via multi-objective loss functions. However, those approaches typically leverage mean field updates based on pairwise distances, which makes them susceptible to cycling behaviors and increased redundancy. In addition, explicitly boosting diversity often has a detrimental impact on optimizing already fruitful behaviors for rewards. As such, the reward-diversity trade off typically relies on heuristics. Finally, such methods require behavioral representations, often handcrafted and domain specific. In this paper, we introduce an approach to optimize all members of a population simultaneously. Rather than using pairwise distance, we measure the volume of the entire population in a behavioral manifold, defined by task-agnostic behavioral embeddings. In addition, our algorithm Diversity via Determinants (DvD), adapts the degree of diversity during training using online learning techniques. We introduce both evolutionary and gradient-based instantiations of DvD and show they effectively improve exploration without reducing performance when better exploration is not required.
Subfields: Artificial Intelligence, Management Information Systems, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Management Information Systems
In Reinforcement Learning
Citations in the corpus
- Dueling RL: Reinforcement Learning with Trajectory Preferences (2021, background): “Many practically relevant trajectory or state-action embeddings can be found defined in Pacchiano et al. (2020); Parker-Holder et al. (2020).”
- Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Games (2021, unknown): “…the geometric perspective that the determinant of the kernel matrix represents the volume of a parallelepiped spanned by feature maps, DvD [25] proposed the concept of population diversity using the determinant of the kernel matrix composed by the behavioral embeddings by multiple…”