Generative Adversarial Imitation from Observation

By Faraz Torabi, Garrett Warnell, Peter Stone. 2018

280 citations recorded by the source corpus.

Imitation from observation (IfO) is the problem of learning directly from state-only demonstrations without having access to the demonstrator's actions. The lack of action information both distinguishes IfO from most of the literature in imitation learning, and also sets it apart as a method that may enable agents to learn from a large set of previously inapplicable resources such as internet videos. In this paper, we propose both a general framework for IfO approaches and also a new IfO approach based on generative adversarial networks called generative adversarial imitation from observation (GAIfO). We conduct experiments in two different settings: (1) when demonstrations consist of low-dimensional, manually-defined state features, and (2) when demonstrations consist of high-dimensional, raw visual data. We demonstrate that our approach performs comparably to classical imitation learning approaches (which have access to the demonstrator's actions) and significantly outperforms existing imitation from observation methods in high-dimensional simulation environments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “However, since equivalence in single state distribution does not mean equivalence in policy, getting the state-transition distributions closer together may result in better performance [Torabi et al., 2018b; 2019a; 2019b].”
Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning (2018, methodology): “The contributions in this work are complementary (and compatible) to Torabi et al. (2018).”
Imitation Learning as f-Divergence Minimization (2019, background): “This connection to GANs leads to interesting avenues such as stabilizing min-max games [36], learning from pure observations [37–39] and links to f-divergence minimization [6, 40].”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, methodology): “Henderson et al. (2018) and Torabi et al. (2018b) extend Generative Adversarial Imitation Learning (Ho & Ermon, 2016) to remove the need for action labels.”
A Survey of Imitation Learning Methods, Environments and Metrics (2024, methodology): “Ho and Ermon’s work [44] is the most known work and perhaps one of the most influential, with several methods following the same setting [57, 70, 79, 103].”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “cloning [55,8,62,11,76] and inverse reinforcement learning [64,49,1,31,23,5,77,45], but also augmenting the RL training with demonstrations [54,21,78,60,58].”
XIRL: Cross-embodiment Inverse Reinforcement Learning (2021, background): “Many recent works [28, 29, 30, 31] build upon GAIL in order to perform observation-only imitation learning using state-only occupancy matching [28], domain adaptation via domain confusion [29, 30], and state-alignment using a variational autoencoder next-state predictor [31].”
Mo' States Mo' Problems: Emergency Stop Mechanisms from Observation (2019, background): “This typically requires action observations [13] or expert feedback [20], although recent work has studied more relaxed settings [24].”
Adversarial Imitation Learning from State-only Demonstrations (2019, unknown): “∗This is an abbreviated version of the work presented in [13].”
AMP (2021, background): “…to settings with state-only demonstrations, the discriminator can be trained on state transitions 𝐷 ( s , s ′ ) instead of state-action pairs 𝐷 ( s , a ) [Torabi et al. 2018], (5) M ( s , s ′ ) and 𝑑 𝜋 ( s , s ′ ) denote the likelihood of observing a state transition from s to s ′ in the…”
SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments (2021, background): “While several prior works have studied imitation without example actions (Liu et al., 2018; Torabi et al., 2018a; Aytar et al., 2018; Torabi et al., 2018b; Edwards et al., 2018; Lee et al.)”
Imitation Learning: A Survey of Learning Methods, Environments and Metrics (2024, unknown): “However, some work [5, 16, 20, 22, 49, 50, 68] use only validation environments, which can be trouble-some.”
A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges (2023, unknown): “Enabling agents to learn from the demonstrations without explicit action information unlocks a wealth of previously inaccessible resources, such as online videos [88].”
Recent Advances in Imitation Learning from Observation (2019, methodology): “To resolve this issue, the authors [Torabi et al., 2018b; Torabi et al., 2019a] proposed an algorithm similar to those above, but with the difference that the discriminator considers state transitions, {(st, st+1)}, as the input instead of single states.”
Imitation Learning from Video by Leveraging Proprioception (2019, methodology): “In Figure 2, we can see that two of the baseline methods— BCO and TCN—do not achieve task performance anywhere near that of the expert.”