Reinforcement Learning as One Big Sequence Modeling Problem

By Michael Janner, Qiyang Li, Sergey Levine. 2021

93 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, methodology): “…‘environment forcing’ (Section 4.2) is inspired by techniques for minimizing exposure bias [33]–[35] and corresponds to the conditioning on sequences of prior actions and states undertaken by [17], [18] when using ‘decision transformers’ and ‘trajectory transformers’ in imitation learning contexts.”
Shaking the foundations: delusions in sequence models for interaction and control (2021, methodology): “More precisely, given past interactions a1:t and o1:t, can we use this model for iteratively choosing the next action At+1 by sampling it from at+1 ∼ P(At+1 | a1:t, o1:t), where a1:t are the previous actions generated by the model and o1:t are the past observations produced by the environment? This ”
Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, methodology): “[44] propose Trajectory Transformer, which is similar to Decision Transformer but additionally uses state and return prediction, as well as discretization, which incorporates model-based components.”
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, methodology): “They even struggled with a diagnostic dataset with paired good and bad human demonstration trajectories while IRIS was able to improve slightly on BC-RNN, suggesting that combining history-dependence with value learning might be a good place to start for improving batch RL methods [48–51].”
Boosting Search Engines with Interactive Agents (2021, neutral): “We have started to investigate approaches in the direction of decision/trajectory transformers (Chen et al., 2021; Janner et al., 2021).”
Can Wikipedia Help Offline Reinforcement Learning? (2022, methodology): “Adopting pre-training as a default option for recent Transformer-based methods (Chen et al., 2021; Janner et al., 2021; Furuta et al., 2021) appears far away – if we only look within RL. Uniﬁed under the umbrella of sequence modeling, we look at whether Transformer-based pre-trained language models…”
RT-1: Robotics Transformer for Real-World Control at Scale (2022, background): “, 2019), locomotion (Janner et al., 2021; Gupta et al., 2022), and manipulation (Jiang et al.”
Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory (2022, background): “This parallel characteristic makes Transformer an ideal model for reinforcement learning [16, 5].”
Policy Gradients Incorporating the Future (2021, methodology): “This is inspired by work that models trajectories using a transformer [16] for offline RL.”