Coordinated Exploration in Concurrent Reinforcement Learning

By Maria Dimakopoulou, Benjamin Van Roy. 2018

46 citations recorded by the source corpus.

We consider a team of reinforcement learning agents that concurrently learn to operate in a common environment. We identify three properties - adaptivity, commitment, and diversity - which are necessary for efficient coordinated exploration and demonstrate that straightforward extensions to single-agent optimistic and posterior sampling approaches fail to satisfy them. As an alternative, we propose seed sampling, which extends posterior sampling in a manner that meets these requirements. Simulation results investigate how per-agent regret decreases as the number of agents grows, establishing substantial advantages of seed sampling over alternative exploration schemes.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chongjie Zhang
2. Tonghan Wang
3. Shimon Whiteson
4. Jianhao Wang
5. Tabish Rashid
6. Anuj Mahajan
7. Yi Wu
8. Chao Yu

In Management Science and Operations Research

1. Shimon Whiteson
2. Chongjie Zhang
3. Tonghan Wang
4. Tabish Rashid
5. Jianhao Wang
6. Yi Wu
7. Shamim Nemati
8. Chao Yu

Citations in the corpus

• Reinforcement Learning in Healthcare: A Survey (2019, background): “proposed, such as PAC guaranteed exploration methods targeting at continuous spaces [150], [151], concurrent exploration mechanisms [318], [319], [320] and exploration in deep RL [321], [322], [323].”
• MAVEN: Multi-Agent Variational Exploration (2019, background): “[7] consider the setting of concurrent RL in which multiple agents interact with their own environments in parallel.”
• Influence-Based Multi-Agent Exploration (2019, methodology): “Dimakopoulou & Van Roy (2018) and Dimakopoulou et al. (2018) study coverage problems in the concurrent reinforcement learning, Bargiacchi et al. (2018) propose an exploration method in repeated single-stage problems, and Iqbal & Sha (2019) study several methods to combining de-centralized curiosity.”