Reinforcement Learning: An Introduction

By Richard S. Sutton, Andrew Barto. 1998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Playing Atari with Deep Reinforcement Learning (2013, background): “Perhaps the most similar prior work to our own approach is neural ﬁtted Q-learning (NFQ) [20].”
Unified Scaling Laws for Routed Language Models (2022, methodology): “We use REINFORCE with a learned baseline [Williams, 1992, Sutton and Barto, 2018] to address this issue, so that improving the policy means increasing the likelihood of selecting experts which lead to a better than average next token prediction.”
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021, background): “As in reinforcement learning, a classic explorationexploitation dilemma arises (Sutton and Barto, 2018).”
Augmented Language Models: a Survey (2023, background): “Such data cannot be used in a supervised setting, but can provide rewards in the context of Reinforcement Learning (RL) (Sutton and Barto, 2018).”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, background): “Although known to be crucial [42], exploration is surprisingly understudied when RL is applied to text generation.”
Investigating Contingency Awareness Using Atari 2600 Games (2012, methodology): “An -greedy policy (Sutton and Barto 1998) is used, which takes exploratory actions Figure 3: Average score over the last 500 episodes for each of the training games.”
The Arcade Learning Environment: An Evaluation Platform for General Agents (2012, methodology): “A detailed explanation of SARSA(λ) and its extensions can be found in the work of Sutton and Barto (1998).”
OpenAI Gym (2016, background): “In the RL literature, the environment is formalized as a partially observable Markov decision process (POMDP) [12].”
Asynchronous Methods for Deep Reinforcement Learning (2016, methodology): “The target value used by one-step Sarsa is r + γQ ( s (cid:48) , a (cid:48) ; θ − ) where a (cid:48) is the action taken in state s (cid:48) (Rummery & Niranjan, 1994; Sutton & Barto, 1998).”
From Pixels to Torques: Policy Learning with Deep Dynamical Models (2015, background): “A promising approach toward solving the pixels to torques problem is Reinforcement Learning (RL) (Sutton & Barto, 1998), a principled mathematical framework that deals with fully autonomous learning from trial and error.”
One-Shot Imitation Learning (2017, background): “Reinforcement learning (Sutton & Barto, 1998; Bertsekas & Tsitsiklis, 1995) provides an alternative route to skill acquisition, by learning through trial and error.”
Artificial Intelligence, Values, and Alignment (2020, background): “These approaches evaluate the ‘lifetime’ behaviour of many agents, each using a different policy for interacting with its environment, and select those behaviours that are able to obtain the most overall reward (Sutton and Barto, 2018, 6).”
Scalable agent alignment via reward modeling: a research direction (2018, methodology): “COACH (MacGlashan et al., 2017; Arumugam et al., 2018) is an algorithm that trains a policy from feedback in the form of an advantage function (Sutton & Barto, 2018).”
Reward learning from human preferences and demonstrations in Atari (2018, background): “The agent learns an estimate of the action-value function (Sutton and Barto, 2018) Q(o, a), approximated by a deep neural network with parameters θ that outputs a set of action-values Q(o, ·; θ) for a given input observation o.”
Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “The RL algorithm we use is a batched synchronous advantage actor-critic (A2C; Mnih et al. (2016)), or equivalently REINFORCE with baseline (Sutton and Barto, 2018); we found that V-MPO (Song et al., 2019) did not improve performance significantly and is computationally more expensive.”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “This setting is equivalent to a bandit feedback environment where the action space is the space of all possible generations for the task (Sutton & Barto, 2018).”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, methodology): “Overall, our contributions are the following: • We clarify relations between the RM and DM paradigms through a detailed comparison between the family of DPG algorithms and Policy Gradients (Table 1), stressing the differences between parametric and non-parametric rewards that are important in this r”
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, background): “[36] use reinforcement learning [132] to generate prompts to control the text generation process.”