Function Optimization using Connectionist Reinforcement Learning Algorithms

By Ronald J. Williams, Jing Peng. 1991

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “This technique was proposed as early as the first work on the REINFORCE algorithm (Williams and Peng, 1991; Williams, 1992), and is often used in recent methods (see, e.g., discussion by O’Donoghue et al. (2017)).”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “The key idea of including a maximum entropy regularizer to encourage exploration is common in RL (Williams & Peng, 1991; Mnih et al., 2016) and inverse RL (Ziebart et al., 2008).”
Towards Better Decoding and Language Model Integration in Sequence to Sequence Models (2016, methodology): “Smoothing techniques increase the entropy of a model’s predictions, a technique that was used to promote exploration in reinforcement learning [27, 28, 29].”
Regularizing Neural Networks by Penalizing Confident Output Distributions (2017, methodology): “In reinforcement learning, encouraging the policy to have an output distribution with high entropy has been used to improve exploration (Williams & Peng, 1991).”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, methodology): “Entropy regularization: An additional component we apply to SLBO is the commonly-adopted entropy regularization in policy gradient method (Williams & Peng, 1991; Mnih et al., 2016), which was found to signiﬁcantly boost the performance in our experiments (ablation study in Appendix F.5).”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, background): “These approaches are very closely related to maximum entropy reinforcement learning techniques which add an entropy regularization ‘bonus’ to prevent early convergence to deterministic policies and thereby heuristically encourage exploration [57, 59, 28, 35, 2, 25].”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “Standard regularization techniques are leveraged in supervised learning to address such ill-specified estimation problems, and have been employed in the RL setting as well [12, 50, 62].”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, background): “Another approach is entropy regularization [51, 20, 38, 47], where causal entropy is added to the rewards, making the Q-function smoother and facilitating optimization [2].”
DeepMind Control Suite (2018, methodology): “Due to the diﬀerent parallelization architectures, the evaluation protocol for each agent was slightly diﬀerent: DDPG was evaluated for 10 episodes for every 100000 steps (with no exploration noise), and A3C was trained with 32 workers and concurrently evaluated with another worker that updated its ”
Reward Augmented Maximum Likelihood for Neural Structured Prediction (2016, background): “By contrast, reinforcement learning (RL) advocates optimizing expected reward (with a maximum entropy regularizer [38]), which is formulated as minimization of the following objective,”
Ask the Right Questions: Active Question Reformulation with Reinforcement Learning (2017, methodology): “To compute gradients for training with use the REINFORCE method [Williams and Peng, 1991],”
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning (2017, methodology): “To delay premature convergence we add an entropy term to the objective (Williams and Peng, 1991; Mnih et al., 2016).”
Language GANs Falling Short (2018, background): “…discriminator (Guo et al., 2017), step-level loss instead of sequence level (Fedus et al., 2018), learned baseline to reduce variance (Fedus et al., 2018), regularizing REINFORCE with a maximum-entropy loss (Williams & Peng, 1991) and alternating adversarial loss with MLE loss (Guo et al., 2017).”
Metacontrol for Adaptive Imagination-Based Optimization (2017, methodology): “To discourage the manager from learning what is an essentially deterministic policy, we included a regularization term based on the entropy, L H (Williams & Peng, 1991; Mnih et al., 2016): r is the full return given by (3) and λ is the strength of the regularization term.”
TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments (2018, background): “The reward shaping term is F ( s, a, s (cid:48) ) = φ ( s (cid:48) ) − φ ( s ) , where the potential function φ ( s ) is the length of the shortest path from agent position at the state s to the goal state.”