Reinforcement learning in feedback control

By Roland Hafner, Martin A. Riedmiller. 2011

207 citations recorded by the source corpus.

Subfields: Computational Theory and Mathematics, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Continuous control with deep reinforcement learning (2015, methodology): “NFQCA [4], which uses the same update rules as DPG but with neural network function approximators, uses batch learning for stability, which is intractable for large networks.”
High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015, background): “Another class of policy gradient algorithms, called actor-critic methods, use a value function rather than the empirical returns, obtaining an estimator with lower variance at the cost of introducing bias (Konda & Tsitsiklis, 2003; Hafner & Riedmiller, 2011).”
Memory-based control with recurrent neural networks (2015, methodology): “Similar ideas had previously been exploited in NFQCA [4] and in the ADP [13] community.”
Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies (2015, background): “An alternate approach, based on the idea of backpropagating the gradient of the value function, is developed in (Jordan and Jacobs, 1990; Prokhorov and Wunsch, 1997; Wang and Si, 2001; Hafner and Riedmiller, 2011; Fairbank and Alonso, 2012; Fairbank et al., 2013).”
Data-efficient Deep Reinforcement Learning for Dexterous Manipulation (2017, methodology): “NFQ [29] and NFQCA [9] employ the same basic update as DDPG and DQN, however, they are batch algorithms that perform updates less frequently and fully re-ﬁt the Q-function and the policy network after every episode with several hundred iterations of gradient descent with Rprop [28] and using…”
Reinforcement Learning with Deep Energy-Based Policies (2017, methodology): “…the connection between our approach and the deep deterministic policy gradient method (DDPG) (Lillicrap et al., 2015), which updates a Q-function critic according to (hard) Bellman updates, and then backpropagates the Q-value gradient into the actor, similarly to NFQCA (Hafner & Riedmiller, 2011).”
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2016, methodology): “This method is closely related to Deep Deterministic Policy Gradient (DDPG) [10] as well as NFQCA [31], with principle differences being that NFQCA employs a batch episodic update and typically resets network parameters between episodes.”
Continuous Deep Q-Learning with Model-based Acceleration (2016, methodology): “…have been applied to learn policies to play Atari games (Mnih et al., 2015; Schaul et al., 2015) and perform a wide variety of simulated and real-world robotic control tasks (Hafner & Riedmiller, 2011; Lillicrap et al., 2016; Levine & Koltun, 2013; de Bruin et al., 2015; Hafner & Riedmiller, 2011).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “While the reinforcement learning algorithms have been an active area of research for decades, the introduction of effective high-capacity function approximators – deep neural networks – into reinforcement learning, along with effective algorithms for training them, has allowed reinforcement learning”
Counterfactual Multi-Agent Policy Gradients (2017, background): “…REINFORCE (Williams, 1992), in which the gradient is: (1) In actor-critic approaches (Kimura et al., 2000; Schulman et al., 2015; Wang et al., 2016; Hafner & Riedmiller, 2011), the actor , i.e., the policy, is trained by following a gradient that depends on a critic , which usually estimates a…”
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation (2018, methodology): “Prior work has sought to address this by using a second network that amortizes the maximization [15, 16], or constraining the Q-function to be convex in a, making it easy to maximize analytically [29, 35].”
Distributed Distributional Deterministic Policy Gradients (2018, methodology): “The reward is given by r “ ψ p palm height ´ obj height ; c, m q where ψ p (cid:15) ; c, m q is a soft indicator function similar to one described by Hafner & Riedmiller (2011) ψ p (cid:15) ; c, m q “ 1 ´ tanh p q 2 if (cid:15) ą c, 1 otherwise.”
Connecting Generative Adversarial Networks and Actor-Critic Methods (2016, methodology): “We focus on deterministic policy gradients (DPG) [7, 10] and its extension to stochastic policies, SVG(0) [8], as well as neurally-fitted Q-learning with continuous actions (NFQCA) [9].”
Deep Reinforcement Learning in Large Discrete Action Spaces (2015, methodology): “DPG is similar to work introduced by NFQCA (Hafner & Riedmiller, 2011) and leverages the gradient-update originally introduced by ADHDP (Prokhorov et al., 1997).”
Reinforcement learning for control: Performance, stability, and deep approximators (2018, background): “3A different version of NFC for continuous actions (NFQ-CA) does exist (Hafner and Riedmiller, 2011).”
Cooperation and communication in multiagent deep reinforcement learning (2016, methodology): “are gradients with respect to inputs, first used in this way by NFQCA (Hafner and Riedmiller, 2011).”
Neuroscience-Inspired Artificial Intelligence. (2017, unknown): “From their neuroscience-informed origins, TD methods and related techniques have gone on to supply the core technology for recent advances in AI, ranging from robotic control (Hafner and Riedmiller, 2011) to expert play in backgammon (Tesauro, 1995) and Go (Silver et al., 2016).”