Neural Fitted Q Iteration - First Experiences with a Data Efficient Neural Reinforcement Learning Method

By Martin A. Riedmiller. 2005

1,097 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Playing Atari with Deep Reinforcement Learning (2013, methodology): “Since Q maps historyaction pairs to scalar estimates of their Q-value, the history and the action have been used as inputs to the neural network by some previous approaches [20, 12].”
Asynchronous Methods for Deep Reinforcement Learning (2016, methodology): “By storing the agent’s data in an experience replay memory, the data can be batched (Riedmiller, 2005; Schulman et al., 2015a) or randomly sampled (Mnih et al., 2013; 2015; Van Hasselt et al., 2015) from different time-steps.”
Learning Continuous Control Policies by Stochastic Value Gradients (2015, background): “It is also possible to backpropagate analytic action derivatives from a Q-function to compute the policy gradient without a model [31, 21, 23].”
Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, neutral): “[10, 49]).”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “…learning differ from our work: They either have focused on scenarios where the policy is able to explore and collect more data (Degris et al., 2012; Riedmiller, 2005) such as learning online from an outdated replay buffer (e.g. (Munos et al., 2016)), or have performed off-policy policy evaluation…”
Data-efficient Deep Reinforcement Learning for Dexterous Manipulation (2017, methodology): “NFQ [29] and NFQCA [9] employ the same basic update as DDPG and DQN, however, they are batch algorithms that perform updates less frequently and fully re-fit the Q-function and the policy network after every episode with several hundred iterations of gradient descent with Rprop [28] and using…”
PGQ: Combining policy gradient and Q-learning (2016, background): “Recently, the ﬁeld of deep reinforcement learning, i.e. , the use of deep neural networks to represent action-values or a policy, has seen a lot of success (Mnih et al., 2015; 2016; Silver et al., 2016; Riedmiller, 2005; Lillicrap et al., 2015; Van Hasselt et al., 2016).”
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates (2016, methodology): “In this work, we focus on model-free RL, which includes policy search methods [24], [3], [23] and function approximation methods [25], [26], [14].”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “reuse it multiple times to perform reinforcement learning updates (Riedmiller, 2005).”
When to use parametric models in reinforcement learning? (2019, background): “For instance, DQN [Mnih et al., 2013, 2015] and neural-fitted Q-iteration [Riedmiller, 2005] match Algorithm 1, if we stretch the definitions of ‘model’ to include the more limited replay buffers.”
Distributed Prioritized Experience Replay (2018, methodology): “It is particularly useful when training neural network function approximators with stochastic gradient descent algorithms, as in Neural Fitted Q-Iteration (Riedmiller, 2005) and Deep Q-Learning (Mnih et al., 2015).”
Deep Reinforcement Learning with Double Q-Learning (2015, methodology): “This is important because ﬂexi-ble parametric function approximators are often employed in reinforcement learning (see, e.g., Tesauro 1995; Sallans and Hinton 2004; Riedmiller 2005; Mnih et al. 2015).”
Learning Combinatorial Optimization Algorithms over Graphs (2017, methodology): “2 Learning algorithm In order to perform end-to-end learning of the parameters in Q̂(h(S), v; Θ), we use a combination of n-step Q-learning [36] and fitted Q-iteration [33], as illustrated in Algorithm 1.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Another commonly used term is “batch reinforcement learning” (Ernst et al., 2005; Riedmiller, 2005; Lange et al., 2012).”
Deep learning in neural networks: An overview (2014, methodology): “More recently, a rather deep GPU-CNN was used in a traditional RL framework to play several Atari 2600 computer games directly from 84 × 84 pixel 60 Hz video input (Mnih et al., 2013), using experience replay (Lin, 1993), extending previous work on Neural Fitted Q-Learning (NFQ) (Riedmiller, 2005).”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, methodology): “In order to test this hypothesis, we use a variant of the neural fitted Q-learning framework [27] with learned keypoints as input and a recurrent neural network Q function to output behaviors.”
Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “, 2005) and neural networks (Riedmiller, 2005), but come without convergence guarantees.”
Deconfounding Reinforcement Learning in Observational Settings (2018, methodology): “In order to reduce non-stationarity and to decorrelate updates, the generated data is stored in an experience replay memory and then randomly sampled in a batch manner (Mnih et al., 2013; Riedmiller, 2005; Schulman et al., 2015; Van Hasselt et al., 2016).”
Fitted Q-Learning for Relational Domains (2020, methodology): “A set of classical approaches for this problem based on Approximate Dynamic Programming (ADP) is the fitted value iteration algorithm (Boyan and Moore 1995; Ernst, Geurts, and Wehenkel 2005; Riedmiller 2005), a batch mode approximation scheme that employs function approximators in each iteration to ”
Reinforcement Learning in Healthcare: A Survey (2019, methodology): “There are several famous BRL approaches in the literature, such as the non-linear approximator cases of Neural Fitted Q Iteration (NFQI [40]), the Tree-based FQI”