A Menu of Designs for Reinforcement Learning Over Time

By W. Miller, Richard S. Sutton, Paul J. Werbos. 1995

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Learning Continuous Control Policies by Stochastic Value Gradients (2015, background): “It is also possible to backpropagate analytic action derivatives from a Q-function to compute the policy gradient without a model [31, 21, 23].”
Data-efficient Deep Reinforcement Learning for Dexterous Manipulation (2017, background): “The use of rich and flexible function approximators such as neural networks in RL dates back many years, e.g. [38, 35, 12, 10].”
Distributed Prioritized Experience Replay (2018, methodology): “(2014) also similar to older methods (Werbos, 1990; Prokhorov & Wunsch, 1997), and tested it on continuous control tasks from the DeepMind Control Suite (Tassa et al.”
RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “en either sensitivity analysis or contribution analysis may be utilized. The best known backward view approach is sensitivity analysis (computing the gradient) like ”Backpropagation through a Model´´ [86, 101, 102, 142, 5]. Sensitivity analysis has several drawbacks: local minima, instabilities, exp”
Model-free Q-learning designs for discrete-time zero-sum games with application to H-infinity control (2007, methodology): “Approximate dynamic programming was proposed by Werbos (1991), Barto, Sutton, and Anderson (1983), Howard (1960), Watkins (1989), Bertsekas and Tsitsiklis (1996), and This paper was not presented at any IFAC meeting.”
Reinforcement Learning in Continuous Time and Space (2000, background): “• An efficient control policy can be derived using the gradient of the value function (Werbos, 1990).”
Reinforcement learning and adaptive dynamic programming for feedback control (2009, methodology): “These problems are solved by introducing a second neural network for the control policy, known as the actor NN [Werbos 1974, 1989, 1991, 1992].”
NeuroAnimator: fast neural network emulation and control of physics-based models (1998, background): “Werbos writes in (Werbos, 1990): \Adaptive critic is an approximation to dynamic programming which is the only exact and e cient method available to control motors or muscles over time so as to maximize the utility function in a noisy, nonlinear environment, without making highly specialized assumpt”
Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning (2019, methodology): “imator [29] and has been widely used for RL [30]–[32].”
Discrete-Time Nonlinear HJB Solution Using Approximate Dynamic Programming: Convergence Proof (2008, methodology): “ADP was proposed by Werbos [12], Barto et.”
Continual learning in reinforcement environments (1995, methodology): “This technique has been used both for supervised learning tasks (training the controller to produce speci c results in the environment) [44, 46, 69], and for reinforcementlearning tasks (training the controller to maximize the predicted reinforcement) [45, 68, 91, 93, 116].”
From model-based control to data-driven control: Survey, classification and perspective (2013, background): “Approximate dynamic programming has been proposed in [147,148] as a solution to optimal control problems forward-in-time.”
Adaptive Dynamic Programming for Control: A Survey and Recent Advances (2021, methodology): “Werbos [11] classified ACD into three classes, namely, heuris-”
Value Iteration Adaptive Dynamic Programming for Optimal Control of Discrete-Time Nonlinear Systems (2016, methodology): “Among these approximate methods, adaptive dynamic programming (ADP), proposed by Werbos [18], [19], as an effective data-based approach to solve optimal control problems forward-in-time, has gained much attention from researchers [20]–[29].”
A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems (2013, methodology): “Owing to the similarities between optimal control and RL (Sutton, Barto, & Williams, 1992), Werbos (1990) introduced RL-based actor–critic methods for optimal control, called Approximate Dynamic Programming (ADP).”