Hybrid Reinforcement Learning with Expert State Sequences
By Xiaoxiao Guo, Shiyu Chang, Mo Yu, G. Tesauro, Murray Campbell. 2019
36 citations recorded by the source corpus.
Existing imitation learning approaches often require that the complete demonstration data, including sequences of actions and states, are available. In this paper, we consider a more realistic and difficult scenario where a reinforcement learning agent only has access to the state sequences of an expert, while the expert actions are unobserved. We propose a novel tensor-based model to infer the unobserved actions of the expert state sequences. The policy of the agent is then optimized via a hybrid objective combining reinforcement learning and imitation learning. We evaluated our hybrid approach on an illustrative domain and Atari games. The empirical results show that (1) the agents are able to leverage state expert sequences to learn faster than pure reinforcement learning baselines, (2) our tensor-based action inference model is advantageous compared to standard deep neural networks in inferring expert actions, and (3) the hybrid policy optimization objective is robust against noise in expert state sequences.
Subfields: Artificial Intelligence, Computational Mathematics, Statistical and Nonlinear Physics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Faraz Torabi
- Peter Stone
- Ruohan Zhang
- Garrett Warnell
- Odinaldo Rodrigues
- Michael Luck
- Nathan Gavenski
- Lin Guan
In Statistical and Nonlinear Physics
- Peter Stone
- Faraz Torabi
- Garrett Warnell
- Felipe Meneguzzi
- Odinaldo Rodrigues
- Michael Luck
- Nathan Gavenski
- Ruohan Zhang
Citations in the corpus
- Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “If a sparse reward function is available, this algorithm can be combined with an RL algorithm to improve the performance [ Guo et al. , 2019 ] .”
- A Survey of Imitation Learning Methods, Environments and Metrics (2024, background): “Given these improvements, imitation learning has seen many applications, including robotics [35, 59, 64, 73, 76, 77, 82, 97, 117, 120], game-playing [5, 29, 38, 42, 55, 69, 74, 77, 79, 87, 93, 94, 119, 122], and natural language processing [27, 56, 66, 81, 111].”
- Recent Advances in Imitation Learning from Observation (2019, background): “While there is some previous work that considers a combination of imitation learning and RL [Zhu et al. , 2018] or IfO and RL [Guo et al. , 2019], there is still much to investigate.”