Robust Deep Reinforcement Learning with Adversarial Attacks

By Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, Girish V. Chowdhary. Adaptive Agents and Multi-Agent Systems, 2017

366 citations recorded by the source corpus.

This paper proposes adversarial attacks for Reinforcement Learning (RL). These attacks are then leveraged during training to improve the robustness of RL within robust control framework. We show that this adversarial training of DRL algorithms like Deep Double Q learning and Deep Deterministic Policy Gradients leads to significant increase in robustness to parameter variations for RL benchmarks such as Mountain Car and Hopper environment. Full paper is available at (https://arxiv.org/abs/1712.03632)[32]

Subfields: Artificial Intelligence, Automotive Engineering

View Paper: https://www.semanticscholar.org/paper/3b6c891fbccaa564ea4fd8914a5e3952fcf42ee3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bo Li
2. Cho-Jui Hsieh
3. Quoc V. Le
4. Yang Liu
5. Huan Zhang
6. Trevor Darrell
7. Sergey Levine
8. Percy Liang

In Automotive Engineering

1. Bo Li
2. Sergey Levine
3. Trevor Darrell
4. Raquel Urtasun
5. Binh Yang
6. Yin Zhou
7. Oncel Tuzel
8. Wenjie Luo

Citations in the corpus

• Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Neural Network Robustness Verification (2021, background): “Finally, we have only demonstrated robustness verification of image classification tasks, and generalizing it to give verification guarantees for other tasks such as robust deep reinforcement learning [27, 48, 49] is an interesting direction for future work.”
• Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges (2021, background): “• Adversarial RL [97], [178] adopts a two-player game structure where the learner agent learns to take actions against an adversarial agent whose objective is different from or even opposite to the learner agent.”
• Regularization Matters in Policy Optimization (2019, background): “, 2017), parameter variations (Pattanaik et al., 2018), or different random seeds in environment generation (Zhang et al.”
• A survey of multi-agent deep reinforcement learning with communication (2022, neutral): “Robustness has been widely considered in the field of reinforcement learning [119, 120], where an agent needs to cope with disturbances in learning in order to achieve a robust policy that can generalize under changes in training/test data.”
• Adversarial Policies: Attacking Deep Reinforcement Learning (2019, methodology): “Prior work has also applied adversarial training to improve the robustness of deep RL policies, where the adversary exerts a force vector on the victim or varies dynamics parameters such as friction (Pinto et al., 2017; Mandlekar et al., 2017; Pattanaik et al., 2018).”
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations (2020, methodology): “The formulation in [50] has a glitch that the gradient is evaluated as∇skQ(s, π(s)) rather than∇skQ(s, π(s)).”
• Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learning (2020, methodology): “(Pattanaik et al. 2018) designed methods of generating adversarial examples by maximizing the probability of taking the worst action.”
• Adversarial Deep Reinforcement Learning in Portfolio Management (2018, background): “, who train two reinforcement learning agents and adversarial playing for enhancing main player robustness [20], and L∞ control, we propose adversarial training, which is to adding random noise in market prices.”
• Recent Advances in Adversarial Training for Adversarial Robustness (2021, background): “, 2019], audios [Pandey and Wang, 2018], and reinforcement learning [Pattanaik et al., 2017].”