Principled Methods for Advising Reinforcement Learning Agents

By Eric Wiewiora, Garrison W. Cottrell, Charles Peter Elkan. 2003

238 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

RUDDER: Return Decomposition for Delayed Rewards (2018, background): “Reward redistribution is related to “look-back advice” 69 [65] which, in contrast to reward redistribution, still requires the original MDP for learning.”
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning (2017, background): “(1999) and Wiewiora et al. (2003) outline potential-based terms that realize sufficient conditions for safe shaping.”
Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Tasks (2020, background): “Wiewiora et al. (2003) further extended PBRS to include shaping functions of actions as well as states and Devlin & Kudenko (2012) allow time-varying shaping functions.”
Agent-Agnostic Human-in-the-Loop Reinforcement Learning (2017, background): “Similarly, Wiewiora et al. [48] extend potential shaping to potential-based advice functions, which identiﬁes a similar class of shaping functions on ( s, a ) pairs.”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, background): “Vanilla PBRS was extended by [54], which formulated the potential as a function of both state and actions.”
Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models (2020, background): “It does this by shaping the reward function to incorporate user demonstrations in the form of advice [6] that biases the optimization process towards areas of the state-action space that the demonstrator deems highvalue, without biasing the learned policy away from the optimal solution.”
Model-augmented Prioritized Experience Replay (2022, methodology): “Since there are various proposed reward shaping methods (Pathak et al., 2017; Stadie et al., 2015; Devlin & Kudenko, 2012; Zou et al., 2019; Hu et al., 2020; Wiewiora et al., 2003; Ng et al., 1999), we combine MaPER to some of them.”
Learning from Demonstration for Shaping through Inverse Reinforcement Learning (2016, background): “An extensively explored way to provide denser information is through shaping the existing reward function by adding an extra signal, thus providing the agent with a composite reward [14, 6, 4, 8, 25].”
Theoretical considerations of potential-based reward shaping for multi-agent systems (2011, background): “It is also the authors’ expectation that potential-based advice [8], an extension of potential-based reward shaping to include heuristics based on actions as well as states, could similarly be extended to guarantee consistent Nash Equilibria when applied to multi-agent reinforcement learning.”
Transfer in Reinforcement Learning via Shared Features (2012, background): “As such it refers to a training technique, not a learning mechanism (see Skinner, 1938). a small change in action selection, as an initial state-action value function (Wiewiora et al., 2003).”
Autonomous Agents and Multi-agent Systems Manuscript No. Learning Potential Functions and Their Representations for Multi-task Reinforcement Learning (unknown): “[87,88] introduced shaping functions of the form Fxa x′a′ = γΦ(x′, a′)−Φ(x, a), where a′ is defined as in the learning rule.”
Potential-based difference rewards for multiagent reinforcement learning (2014, methodology): “Furthermore, F (s, s) could be replaced with the more general F (s, a, s, a) to include reward shaping based on actions as well as states [19].”
Reinforcement learning from simultaneous human and MDP reward (2012, background): “As discussed in Section 3.1, Q augmentation is nearly equivalent to a form of reward shaping called look-ahead advice [20].”
FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback (2020, methodology): “ior was, in the opinion of the operator. Potential-based reward shaping (PBRS) methods used ‘potential functions’ to accelerate the learning process, while preserving the identity of optimal policies [31, 47, 49]. The potential function was designed to encode ‘rules’ of the environment of the RL age”
Learning Curriculum Policies for Reinforcement Learning (2018, methodology): “We use potential-based advice [27], which restricts the form of f to be a difference of potential functions:”
Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search (2020, background): “For instance, reward shaping [50, 79, 16, 7, 74] leverages the expertise knowledge to reshape the reward distributions to steer the agent’s action selection and navigation towards the expected trajectories; [59, 84, 39, 31, 49, 38, 74] make use of external demonstrations from different sources such…”
Expressing Arbitrary Reward Functions as Potential-Based Advice (2015, methodology): “R, one may use the biased greedy action-selection (Wiewiora, Cottrell, and Elkan 2003) w.r.t. the initial values of the dynamic advice function. π(s) = arg max a ( Q(s, a) + Φ0(s, a) ) (17) Notice that when the advice function is initialized to 0, the biased greedy action-selection above reduces to…”
Policy Transfer using Reward Shaping (2015, background): “This result was extended to shaping over state-action pairs (s, a) by Wiewiora et al. [25] : F (s, a, s , a ) = γΦ(s , a ) − Φ(s, a) This allows for the incorporation of more specific information , pertaining to both states and actions.”
Reinforcement Learning from Demonstration through Shaping (2015, methodology): “The definition of F and Φ was extended by [Wiewiora et al., 2003] to include actions, allowing for the incorporation of behavioural knowledge, reflecting the quality of actions as well as states: F (s, a, s , a ) = γΦ(s , a ) − Φ(s, a) Wiewiora et al. extended Ng's proof and show that this…”