Self-Imitation Learning

By Junhyuk Oh, Yijie Guo, Satinder Singh, Honglak Lee. 2018

307 citations recorded by the source corpus.

This paper proposes Self-Imitation Learning (SIL), a simple off-policy actor-critic algorithm that learns to reproduce the agent's past good decisions. This algorithm is designed to verify our hypothesis that exploiting past good experiences can indirectly drive deep exploration. Our empirical results show that SIL significantly improves advantage actor-critic (A2C) on several hard exploration Atari games and is competitive to the state-of-the-art count-based exploration methods. We also show that SIL improves proximal policy optimization (PPO) on MuJoCo tasks.

Subfields: Developmental and Educational Psychology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Developmental and Educational Psychology

In Reinforcement Learning

Citations in the corpus

Experience Replay for Continual Learning (2018, background): “Reducing this deviation is our motivation for the behavioral cloning component of CLEAR, inspired by work showing the power of policy consolidation [12], self-imitation [23], and knowledge distillation [5].”
Exploration by Random Network Distillation (2018, background): “Indeed, a mean return of about 2,500 can be reliably achieved without special exploration methods (Horgan et al., 2018; Espeholt et al., 2018; Oh et al., 2018).”
P3O: Policy-on Policy-off Policy Optimization (2019, neutral): “For instance, the authors in Oh et al. (2018) report poor results with ACER and prioritized replay as compared to vanilla actor-critic meth-ods.”
TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, positive): “…of the designated player: where u i build_in is represented for the build-in action and I ( i, t ) is an indicator function, which is deﬁned as: Using past good experiences as the supervision has shown great success in many control tasks (Oh et al., 2018; Vinyals et al., 2019; Zha et al., 2021).”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “[27] proposed to exploit past good experience for better exploration in RL and theoretically justified its effectiveness.”
Reward-Conditioned Policies (2019, background): “…learning in the inner loop of a reinforcement learning update, either by imitating a computational expert (e.g., another more local RL algorithm) [7, 20], the best-performing trajectories [25], or by reweighting sub-optimal data to make it resemble samples from a more optimal policy [31, 32].”
Grandmaster level in StarCraft II using multi-agent reinforcement learning (2019, background): “Similar to self-imitation learning 17 , the idea is to update the policy from partial trajectories with better-than-expected returns by bootstrapping when the behaviour policy takes a worse-than-average action (ablation in Fig.”
Exploration in Deep Reinforcement Learning: A Survey (2022, background): “[88] A2C Goals to Explore from Atari: Montezuma Revenge 2500 (A2C 0) Atari images Ac MF D/ D Guo et al.”
BeBold: Exploration Beyond the Boundary of Explored Regions (2020, background): “We might also set up different goals for exploration (Guo and Brunskill, 2019; Oh et al., 2018; Andrychowicz et al., 2017).”
Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (2020, background): “…(Belle-mare et al., 2016; Tang et al., 2017), unsupervised control tasks (Jaderberg et al., 2017), learning to predict environment dynamics (Houthooft et al., 2016; Pathak et al., 2017; Burda et al., 2018), self-imitation (Oh et al., 2018), and self-play (Sukhbaatar et al., 2017; Liu et al., 2019).”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, methodology): “We apply self-imitation learning [84] (SI) to further improve sample efﬁ-ciency because computing the reward with M INE CLIP in the loop makes the training more expensive.”
Dual Policy Distillation (2020, unknown): “Previous work shows that it is promising to learn from imperfect demonstrations [ Hester et al. , 2018 ] and exploiting own past good experiences will help exploration [Oh et al. , 2018].”
Reinforced Self-Training (ReST) for Language Modeling (2023, background): “Self Imitation Learning (SIL) SIL learns a policy for an off-policy actor-critic algorithm where the policy tries to reproduce the good behaviour demonstrated by the agent (Oh et al., 2018).”
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills (2021, methodology): “Prior methods for goal-conditioned supervised learning (Oh et al., 2018; Ding et al., 2019; Ghosh et al., 2019; Sun et al., 2019; Lynch et al., 2019) have a similar motivation: learning “useful” information for control without access to rewards.”
Generative Adversarial Self-Imitation Learning (2018, background): “also outperforms self-imitation learning (‘PPO+SIL’) [19] showing that our generative adversarial approach is more sample-efficient than self-imitation learning.”
Hierarchical reinforcement learning for efficient exploration and transfer (2020, methodology): “To investigate the transfer ability of the algorithm, we train ’HRL-SIL’ subsequently on a set of tasks and compared to ’SIL-EXP’.”