Universal Value Function Approximators

By Tom Schaul, Dan Horgan, Karol Gregor, David Silver. 2015

1,227 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

The Predictron: End-To-End Learning and Planning (2016, background): “Such predictions have been shown to be useful as representations (Schaul & Ring, 2013) and for transfer (Schaul et al., 2015).”
Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, background): “The UVFA architecture (Schaul et al., 2015a) is a factored representation of a continuous set of optimal value functions, combining features of the state with an embedding of the pseudo-reward function.”
Neural Programmer-Interpreters (2015, background): “…been studied in the literature on hierarchical reinforcement learning (e.g., Dietterich (2000); Andre & Russell (2001); Sutton et al. (1999) and Schaul et al. (2015)), imitation and apprenticeship learning (e.g., Kolter et al. (2008) and Rothkopf & Ballard (2013)) and elicitation of options…”
Meta-Gradient Reinforcement Learning (2018, methodology): “To deal with non-stationarity in the value function and policy, we utilise an idea similar to universal value function approximation (UVFA) [Schaul et al., 2015].”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “Zhu et al. (2017b) input both the ﬁrst-person view and the image of the target object to the A3C model, formulating a target-driven navigation problem based on the universal value function approximators (Schaul et al., 2015a).”
Gradient Episodic Memory for Continual Learning (2017, neutral): “The use of task descriptors is similar in spirit to recent work in Reinforcement Learning [Sutton et al., 2011, Schaul et al., 2015], where task or goal descriptors are also fed as input to the system.”
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, methodology): “Value functions have also been generalized to consider goals along with states [21].”
Same State, Different Task: Continual Reinforcement Learning without Interference (2021, background): “A number of previous works have studied transfer in multitask RL settings where the goals within an environment change (Barreto et al. 2016; Schaul et al. 2015).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “We note that here we only touch upon some open problems and refer the reader to Schaul et al. (2018) for more discussion on open problems in continual RL.”
Efficient Lifelong Learning with A-GEM (2018, methodology): “More generally, tasks descriptors have been used in Reinforcement Learning with similar motivations by several others (Sutton et al., 2011; Schaul et al., 2015; Baroni et al., 2017), and it is also a key ingredient in all the zero/few-shot learning algorithms (Lampert et al., 2014; Xian et al.,…”
Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation (2018, background): “Prior work has aimed to achieve similar results by conditioning the policy on a goal or objective indicator [4, 5].”
End-to-End Driving Via Conditional Imitation Learning (2017, background): “[31] proposed a general framework for reinforcement learning with parameterized value functions, shared across states and goals.”
Learning to Act by Predicting the Future (2016, background): “A general framework for sharing value function approximators across both states and goals has been described by Schaul et al. (2015).”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, background): “Our work builds on goal-conditioned RL [17, 18, 19, 20], which has been explored in the context of reward-free learning [21], learning with sparse rewards [18], large scale generalizable imitation learning [22], and hierarchical RL [15].”
Implicit Quantile Networks for Distributional Reinforcement Learning (2018, background): “If we ignore the distributional interpretation for a moment and view each Z τ ( x, a ) as a separate action-value function, this highlights that implicit quantile networks are a type of universal value function approximator (UVFA) (Schaul et al., 2015).”
One-Shot Visual Imitation Learning via Meta-Learning (2017, methodology): “Unlike prior methods that take the task identity [1, 2, 3, 4] or a demonstration [5] as the input into a contextual policy, our approach learns a parameterized policy that can be adapted to different tasks through gradient updates, effectively learning to imitation learn.”
Overcoming Exploration in Reinforcement Learning with Demonstrations (2017, background): “Instead of the standard RL setting, we train agents with parametrized goals, which lead to more general policies [36] and have recently been shown to make learning with sparse rewards easier [1].”
Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research (2018, methodology): “The list of potential improvements is long e.g. Prioritized Experience Replay (Schaul et al., 2015b), distributional RL (Bellemare et al., 2017), entropy-regularized RL (Schulman et al., 2017a), or reverse curriculum generation (Florensa et al., 2017).”
Hindsight Experience Replay (2017, methodology): “Universal Value Function Approximators (UVFA) (Schaul et al., 2015a) is an extension of DQN to the setup where there is more than one goal we may try to achieve.”
FeUdal Networks for Hierarchical Reinforcement Learning (2017, methodology): “For a simple, tabular case (Wiering & Schmidhu-ber, 1997; Schaul et al., 2015), each state can be used as a sub-goal.”