Imitating Latent Policies from Observation

By Ashley D. Edwards, Himanshu Sahni, Yannick Schroecker, Charles Lee Isbell Jr.. 2018

179 citations recorded by the source corpus.

In this paper, we describe a novel approach to imitation learning that infers latent policies directly from state observations. We introduce a method that characterizes the causal effects of latent actions on observations while simultaneously predicting their likelihood. We then outline an action alignment procedure that leverages a small amount of environment interactions to determine a mapping between the latent and real-world actions. We show that this corrected labeling can be used for imitating the observed behavior, even though no expert actions are given. We evaluate our approach within classic control environments and a platform game and demonstrate that it performs better than standard approaches. Code for this work is available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “A combination of policy with this type of dynamics model can be used to predict the next state from a given current state and then train the imitation policy to make the prediction more accurate [Edwards et al. , 2018].”
A Survey of Imitation Learning Methods, Environments and Metrics (2024, methodology): “For example, in some environments, researchers use how many coins [29, 55] the agent collects during an episode or how many goals a team scores [24, 86].”
Learning latent actions to control assistive robots (2021, unknown): “Robots can learn low-dimensional models of states [40], dynamics [49,50], movement primitives [39], trajectories [14], plans [32], policies [17], skills [42], and action representations for reinforcement learning [11].”
XIRL: Cross-embodiment Inverse Reinforcement Learning (2021, background): “To address this issue, several approaches either try to infer expert actions [10, 11, 12] – for example by training an inverse dynamics model on agent interaction data [10] – or employ forward prediction on the next state to imitate the expert without direct action supervision [13].”
Shared Autonomy with Learned Latent Actions (2020, background): “There is also extensive work in learning latent representations for components in RL [4, 6, 31, 20, 9].”
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (2022, background): “Edwards et al. 46 propose to ﬁrst learn a latent policy using unlabeled demonstrations and then map the learned latent actions to real actions using environment interaction.”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “The objective is to enable the agent to anticipate and mimic these latent causes [84].”
Imitation Learning: A Survey of Learning Methods, Environments and Metrics (2024, unknown): “Given these improvements, imitation learning has seen many applications, including robotics [8–17], game-playing [6, 7, 9, 18–28], and natural language processing [29–33].”
AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (2019, unknown): “actions [44], [47], [51], [56], [59], [60].”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “There are promising techniques that can potentially overcome this limitation, such as the Learning-from-Observation (LfO) family of algorithms [123, 122, 115, 31].”
Recent Advances in Imitation Learning from Observation (2019, methodology): “ILPO creates an initial hypothesis for the imitation policy by learning a latent policy π ( z | s t ) that estimates the probability of latent (unreal) action z given the current state s t .”
Genie: Generative Interactive Environments (2024, methodology): “Prior works have used latent actions for imitation from observation (Edwards et al., 2019), planning (Ry-bkin* et al., 2019) and pre-training RL agents (Schmidt and Jiang, 2024; Ye et al., 2022).”
Provable Representation Learning for Imitation Learning via Bi-level Optimization (2020, methodology): “Though some modelbased methods have been proposed (Torabi et al., 2018; Edwards et al., 2018), these methods lack theoretical guarantees.”
Imitation Learning from Video by Leveraging Proprioception (2019, methodology): “Another model-based approach to IfO is imitating latent policies from observation ( ILPO ) [ Edwards et al. , 2019 ] .”
Learning Navigation Subroutines from Egocentric Videos (2019, unknown): “Contemporary works [23, 24, 25] study the problem of learning from state only trajectories, similar to our work here.”
SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments (2021, background): “While several prior works have studied imitation without example actions (Liu et al., 2018; Torabi et al., 2018a; Aytar et al., 2018; Torabi et al., 2018b; Edwards et al., 2018; Lee et al.)”
AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence (2019, background): “There is a long history of fruitful research in imitation learning and learning via observation that demonstrates the beneﬁts of exploiting such data [39, 14, 165, 7, 145, 38, 185, 132, 119, 1].”
Controlling Assistive Robots with Learned Latent Actions (2019, background): “plans [19], policies [20], and skills for reinforcement learning [21].”