Learning Action Representations for Reinforcement Learning

By Yash Chandak, Georgios Theocharous, James E. Kostas, Emma Jordan, Philip S. Thomas. 2019

193 citations recorded by the source corpus.

Most model-free reinforcement learning methods leverage state representations (embeddings) for generalization, but either ignore structure in the space of actions or assume the structure is provided a priori. We show how a policy can be decomposed into a component that acts in a low-dimensional space of action representations and a component that transforms these representations into actual actions. These representations improve generalization over large, finite action sets by allowing the agent to infer the outcomes of actions similar to actions already taken. We provide an algorithm to both learn and use action representations and provide conditions for its convergence. The efficacy of the proposed method is demonstrated on large-scale real-world problems.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “, 2020) or under-explored (Chandak et al., 2019) actions in RL, learn linear dynamics that help exploration (Kim et al.”
Generalization to New Actions in Reinforcement Learning (2020, background): “…Continuous representations of discrete actions have been primarily used to ease learning in large discrete action spaces (Dulac-Arnold et al., 2015; Chandak et al., 2019) or exploiting the shared structure among actions for efﬁcient learning and exploration (He et al., 2015; Tennenholtz & Mannor,…”
Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation (2020, methodology): “Inspired by (Chandak et al., 2019), we model the action selection using Boltzmann distribution for stability during training: where γ is a temperature parameter, and g ( · ) is a function that maps the input into hidden states of the same dimension as action embeddings.”
Shared Autonomy with Learned Latent Actions (2020, background): “There is also extensive work in learning latent representations for components in RL [4, 6, 31, 20, 9].”
Learning latent actions to control assistive robots (2021, unknown): “Robots can learn low-dimensional models of states [40], dynamics [49,50], movement primitives [39], trajectories [14], plans [32], policies [17], skills [42], and action representations for reinforcement learning [11].”
Controlling Assistive Robots with Learned Latent Actions (2019, background): “s, we turn to works that learn latent representations from data. Recent research has learned latent dynamics [13], trajectories [14], plans [15], policies [16], and actions for reinforcement learning [17]. These methods typically leverage autoencoder models [18], [19], which learn a latent space wit”
Know Your Action Set: Learning Action Relations for Reinforcement Learning (2022, background): “In discrete action RL, action representations have enabled learning in large action spaces (Dulac-Arnold et al., 2015; Chandak et al., 2019), transfer learning to a different action space (Chen et al., 2019b), and efﬁcient exploration by exploiting the shared structure among actions (He et al.,…”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, methodology): “We divide prior work in hierarchical learning into two categories: methods that seek to learn both the low-level and high-level policies through active interaction with an environment (Kupcsik et al., 2013; Heess et al., 2016; Bacon et al., 2017; Florensa et al., 2017; Haarnoja et al., 2018a; Nachum”
Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation (2019, background): “To boost sampling efficiency, recent work has proposed to use generative models [10, 11, 12] to prioritize the sampling of more promising states and actions.”
Toward Robust Long Range Policy Transfer (2021, methodology): “Some methods specify actions through the latent representation which is further transformed to the actions of the underlying system (Merel et al. 2019; Burgard, Brock, and Stachniss 2008; Chandak et al. 2019).”