Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives

By Anirudh Goyal, Shagun Sodhani, Jonathan Binas, Xue Bin Peng, Sergey Levine, Yoshua Bengio. 2019

55 citations recorded by the source corpus.

Reinforcement learning agents that operate in diverse and complex environments can benefit from the structured decomposition of their behavior. Often, this is addressed in the context of hierarchical reinforcement learning, where the aim is to decompose a policy into lower-level primitives or options, and a higher-level meta-policy that triggers the appropriate behaviors for a given situation. However, the meta-policy must still produce appropriate decisions in all states. In this work, we propose a policy design that decomposes into primitives, similarly to hierarchical reinforcement learning, but without a high-level meta-policy. Instead, each primitive can decide for themselves whether they wish to act in the current state. We use an information-theoretic mechanism for enabling this decentralized decision: each primitive chooses how much information it needs about the current state to make a decision and the primitive that requests the most information about the current state acts in the world. The primitives are regularized to use as little information as possible, which leads to natural competition and specialization. We experimentally demonstrate that this policy architecture improves over both flat and hierarchical policies in terms of generalization.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

Citations in the corpus

Recurrent Independent Mechanisms (2019, background): “State dependent activation of different primitive policies was also studied in (Goyal et al., 2019b), and the authors showed that they can learn different primitives, but they also consider that only a single primitive can be active at a particular time step.”
Same State, Different Task: Continual Reinforcement Learning without Interference (2021, background): “Many works have focused on generalization properties of RL agents in the MiniGrid environment (Goyal et al. 2020), training on hundreds of levels.”
Toward Robust Long Range Policy Transfer (2021, methodology): “Some methods try to learn a policy that doesn’t highly depend on task-speciﬁc information during pre-training and only acquire task-speciﬁc information when the policy needs to make a critical decision to solve the task (Goyal et al. 2019; Galashov et al. 2019; Goyal et al. 2020).”
Learning to Coordinate Manipulation Skills via Skill Behavior Diversification (2020, methodology): “Hence, instead of learning a task at once, modular approaches (Andreas et al., 2017; Oh et al., 2017; Frans et al., 2018; Lee et al., 2019; Peng et al., 2019; Goyal et al., 2020) suggest to learn reusable primitive skills and solve more complex tasks by recombining the skills.”
Multi-Task Reinforcement Learning with Soft Modularization (2020, methodology): “By utilizing different modules across different tasks, it reduces the interference of gradients on each module and allows better generalization [36, 5, 1, 32, 27, 23, 9, 33, 8].”