Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

By Ronald J. Williams. 2004

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

A Deep Reinforced Model for Abstractive Summarization (2017, methodology): “(2015) have applied the REINFORCE algorithm (Williams, 1992) to train various RNNbased models for sequence generation tasks, leading to significant improvements compared to previous supervised learning methods.”
Neural Variational Inference and Learning in Belief Networks (2014, methodology): “The resulting training procedure for the inference network can be seen as an instance of the RE-INFORCE algorithm (Williams, 1992).”
Deep AutoRegressive Networks (2013, methodology): “We use a Monte Carlo approximation to estimate ∇ θ E [ f ( H i )] where h i is sampled from q ( h i ) (Williams, 1992; Andrad´ottir, 1998).”
Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “…for compatibility with the dataset of preferences under a preference model such as the Bradley-Terry model [5], then ﬁne-tune a language model to maximize the given reward using reinforcement learning algorithms, commonly REINFORCE [44], proximal policy optimization (PPO; [37]), or variants [32].”
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “We optimize it using gradient descent and teacherforcing (Williams, 1992).”
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, methodology): “The gradient updates are computed using vanilla policy gradient (RE-INFORCE) (Williams, 1992), and we use trust-region policy optimization (TRPO) as the meta-optimizer (Schulman et al., 2015).”
Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation (2013, background): “Fiete and Seung (2006) end up proposing a gradient estimator that looks like a correlation between the reward and the perturbation , very similar to that presented in Section 3, and which is a special case of the REINFORCE (Williams, 1992) algorithm.”
Unified Scaling Laws for Routed Language Models (2022, methodology): “We use REINFORCE with a learned baseline (Williams, 1992; Sutton & Barto, 2018) to address this issue, so that improving the policy means increasing the likelihood of selecting experts which lead to a better than average next token prediction.”
Augmented Language Models: a Survey (2023, methodology): “Optimization is done through policy gradient algorithms like REINFORCE (Williams, 1992), PPO and similar variants (Schulman et al.”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, methodology): “The policy is usually learned via vanilla Policy Gradient REINFORCE [50], with the exception of MaliGAN [7], which Another difﬁculty with GANs for discrete sequential data is that discriminators are inaccurate for samples close to the generator distribution modes, as those used for training are…”
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering (2019, methodology): “Policy Gradient To apply these two rewards, we use the REINFORCE algorithm (Williams, 1992) to learn a generation policy p θ deﬁned by the QG model parameters θ .”
A Discrete Hard EM Approach for Weakly Supervised Question Answering (2019, background): “…reward-based algorithms for weak supervision, including beam-based MML (MML which keeps a beam during training), conventional hard EM 8 , REINFORCE (Williams, 1992), iterative ML (Liang et al., 2017; Abolaﬁa et al., 2018) and a family of MAPO (Memory-augmented policy optimization) (Liang et al.,…”
Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering (2019, methodology): “Motivated by the REINFORCE (Williams, 1992) algorithm, we compute the gradient of our objective as”
Learning with Latent Language (2017, methodology): “Policy search DAgger (Ross et al., 2011) is used for pre-training and vanilla policy gradient (Williams, 1992) for concept learning.”
Dialog-based Language Learning (2016, background): “This strategy is derived naturally as the degenerate case one obtains by applying policy gradient [30] in our setting where the policy is fixed (see end of Sec.”