Opponent Modeling in Deep Reinforcement Learning

By He He, Jordan L. Boyd-Graber. 2016

377 citations recorded by the source corpus.

Opponent modeling is necessary in multi-agent settings where secondary agents with competing goals also adapt their strategies, yet it remains challenging because strategies interact with each other and change. Most previous work focuses on developing probabilistic models or parameterized strategies for specific applications. Inspired by the recent success of deep reinforcement learning, we present neural-based models that jointly learn a policy and the behavior of opponents. Instead of explicitly predicting the opponent's action, we encode observation of the opponents into a deep Q-Network (DQN); however, we retain explicit modeling (if desired) using multitasking. By using a Mixture-of-Experts architecture, our model automatically discovers different strategy patterns of opponents without extra supervision. We evaluate our models on a simulated soccer game and a popular trivia game, showing superior performance over DQN and its variants.

Subfields: Artificial Intelligence, Economics and Econometrics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Reinforcement Learning

Citations in the corpus

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension (2017, background): “, 2012) or using reinforcement learning to model opponent behavior (He et al., 2016).”
OpenSpiel: A Framework for Reinforcement Learning in Games (2019, background): “Game Reference(s) Status Backgammon Wikipedia Breakthrough Wikipedia Bridge bidding Wikipedia Coin Game [56] ∼ Connect Four Wikipedia Cooperative Box-Pushing [62] ∼ Chess Wikipedia First-price Sealed-bid Auction Wikipedia Go Wikipedia Goofspiel Wikipedia Havannah Wikipedia Hex Wikipedia ∼ Kuhn poker”
Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games (2017, neutral): “(2) effectively turns into an MDP problem (He et al. 2016): Q∗(s,a) =r(s,a) + λmax θ Q∗(s′,aθ(s ′)), (3) where we drop notation bφ for brevity.”
Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning (2017, methodology): “He et al. (2016) addresses multi-agent learning by explicitly marginalising the opponents strategy using a mixture of experts in the DQN. Unlike our contributions, none of these papers directly aim to address the nonstationarity arising in multi-agent learning.”
Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (2020, methodology): “These algorithm classes are frequently used in the literature either as baselines or building blocks for more complex algorithms [He et al., 2016, Sukhbaatar et al., 2016, Foerster et al., 2016, Raileanu et al., 2018, Jaques et al., 2019, Iqbal and Sha, 2019, Du et al., 2019, Ryu et al., 2020].”
The Hanabi Challenge: A New Frontier for AI Research (2019, background): “signiﬁcant ﬁnancial support for this work that could have inﬂuenced its outcome.”
Actor-Attention-Critic for Multi-Agent Reinforcement Learning (2018, background): “…& Abbeel, 2017; Lowe et al., 2017; Foerster et al., 2016), credit assignment in cooperative settings with global rewards (Rashid et al., 2018; Sunehag et al., 2017; Foerster et al., 2017a), and the failure to take opponent strategies into account when learning agent policies (He et al., 2016).”
Mean Field Multi-Agent Reinforcement Learning (2018, background): “In addressing the non-stationary issue in MARL, various solutions have been proposed including neural-based opponent modeling (He & Boyd-Graber, 2016), policy parameters sharing (Gupta et al., 2017), etc .”
A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning (2017, methodology): “In future work, we will consider maintaining diversity among oracles via loss penalties based on policy dissimilarity, general response graph topologies, environments such as emergent language games [58] and RTS games [96, 84], and other architectures for prediction of behavior, such as opponent mod”
Emergent Complexity via Multi-Agent Competition (2017, background): “This was further explored by Matignon et al. [2012] again in the cooperative setting.”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “Albrecht and Stone [11] presented a thorough survey in this topic and we have found many works that fit into this category in the MDRL setting, some taking inspiration from DRL [169, 170, 171], and others from MAL [172, 173, 64, 174, 175].”
Relational Forward Models for Multi-Agent Learning (2018, neutral): “Despite these recent successes, coordination is still considered a hard problem and several attempts have been made to promote the emergence of coordination by relaxing some assumptions (Foerster et al., 2016; 2017; Sukhbaatar et al., 2016; Raileanu et al., 2018; He et al., 2016a).”
Multi-objective multi-agent decision making: a utility-based analysis and survey (2019, background): “Very recently, single-objective multi-agent RL has received considerable attention as well [30,32,39,55,97,81,109, 130].”
Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence (2019, background): “Other less commonly adopted ideas include predicting the opponent’s behavior [100, 101] and giving the agents information on their opponents [24, 31].”
Robust Multi-Agent Reinforcement Learning via Minimax Deep Deterministic Policy Gradient (2019, background): “Forester et al. (2016b) and He et al. (2016) extended the deep Q-learning to multi-agent setting; Peng et al. (2017a) proposed a centralized policy learning algorithm based on actor-critic policy gradient; Forester et al. (2016a) developed a decentralized multi-agent policy gradient algorithm with…”
A Deep Policy Inference Q-Network for Multi-Agent Systems (2017, background): “• Unlike the previous works [6, 15, 35] only focusing on competitive environments, our models are capable of handling”
Actor-Critic Policy Optimization in Partially Observable Multiagent Environments (2018, methodology): “There are also methods that attempt to model the opponents [36, 30, 4].”
A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning (2020, background): “The literature on opponent modeling, for instance, infers opponents’ behaviors and conditions an agent’s policy on the inferred behaviors of others (He et al., 2016; Raileanu et al., 2018; Grover et al., 2018).”
Autonomous agents modelling other agents: A comprehensive survey and open problems (2017, background): “, 2017); and using opponent features in a neural network to learn expected action utilities (He et al., 2016).”