Simple statistical gradient-following algorithms for connectionist reinforcement learning

By Ronald J. Williams. 1992

Subfields: Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Asynchronous Methods for Deep Reinforcement Learning (2016, methodology): “One example of such a method is the REINFORCE family of algorithms due to Williams (1992).”
Thinking Fast and Slow with Deep Learning and Tree Search (2017, methodology): “We show that this algorithm signiﬁcantly outperforms a variant of the REINFORCE algorithm in learning to play the board game Hex; this version of REINFORCE has recently achieved state-of-the-art play in the related board game Go.”
Visual Programming: Compositional visual reasoning without training (2022, methodology): “While early attempts use off-the-shelf parsers [2], recent methods [9, 10, 12] learn the layout generation model jointly with the neural modules using REINFORCE [30] and weak answer supervision.”
Importance Weighted Autoencoders (2015, methodology): “4) is another algorithm for training recognition networks which reduces stochasticity in the updates by training a third network to predict reward baselines in the context of the REINFORCE algorithm (Williams, 1992). Salakhutdinov & Larochelle (2010) used a recognition network to approximate the”
Neural Approaches to Conversational AI (2018, positive): “cided by the controller on the ﬂy. More complex queries need more steps. Thus, IRN learns a stochastic policy to get a distribution over termination and prediction actions by the REINFORCE algorithm (Williams, 1992), which is described in Sec. 2.3.2 and Eqn. 2.7. Since all the modules of IRN are dif”
Hierarchical Multiscale Recurrent Neural Networks (2016, positive): “e efforts since the standard backpropagation is no longer applicable due to the non-differentiability. Among a few methods for training a neural network with discrete variables such as the REINFORCE (Williams, 1992; Mnih and Gregor, 2014) and the straight-through estimator (Bengio et al., 2013), we ”
Minimum Risk Training for Neural Machine Translation (2015, methodology): “Building on the REINFORCE algorithm proposed by Williams (1992), MIXER allows incremental learning and the use of hybrid loss function that combines both REINFORCE and cross-entropy.”
Deep learning in neural networks: An overview (2014, neutral): “…extended Kalman filter (Feldkamp, Prokhorov, Eagen, & Yuan, 1998; Feldkamp, Prokhorov, & Feldkamp, 2003; Haykin, 2001; Kalman, 1960; Puskorius & Feldkamp, 1994; Williams, 1992b) and an LSTM RNN (Pérez-Ortiz et al., 2003) learned to deal correctly with values of n up to 10 million and more.”
NAS-Bench-101: Towards Reproducible Neural Architecture Search (2019, neutral): “We did, however, successfully train a naive REINFORCE-based (Williams, 1992) controller which simply outputs a multinomial probability distribution at each of the 21 possible edges and 5 operations and samples the distribution to get a new model.”
Learning Sparse Neural Networks through L0 Regularization (2017, background): “While in principle gradient estimators such as the REINFORCE (Williams, 1992) estimator or the straight-through (Bengio et al., 2013) estimator could be employed, they suffer from high variance and biased gradients respectively.”
Reinforcement learning in robotics: A survey (2013, background): “The weight perturbation algorithm is related to REINFORCE but can deal with non-Gaussian distributions which significantly improves the signal-tonoise ratio of the gradient (Roberts et al., 2010).”
Neural Combinatorial Optimization with Reinforcement Learning (2016, methodology): “The gradient of (3) is formulated using the well-known REINFORCE algorithm (Williams, 1992): b i ← b θ v ( s i ) for i ∈ { 1 , . . . , B } 8: end for 13: return θ 14: end procedure where b ( s ) denotes a baseline function that does not depend on π and estimates the expected tour length to reduce…”
Learning to Learn for Global Optimization of Black Box Functions (2016, methodology): “If even at training time the derivatives of f are not available then it would be necessary to approximate these derivatives via an algorithm such as REINFORCE [Williams, 1992].”
Action Recognition using Visual Attention (2015, methodology): “…attention models are deterministic and can be trained using backpropagation, whereas hard attention models are stochastic and can be trained by the REINFORCE algorithm (Williams, 1992; Mnih et al., 2014), or by maximizing a variational lower bound or using importance sampling (Ba et al., 2015b;a).”