Reinforcement learning by reward-weighted regression for operational space control

By Jan Peters, Stefan Schaal. 2007

456 citations recorded by the source corpus.

Subfields: Aerospace Engineering, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Trust Region Policy Optimization (2015, methodology): “We compared the single path and vine algorithms to two other approaches: reward-weighted regression (RWR) [26]5 and the cross entropy method.”
Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “Following prior work [29, 28, 17, 15], it is straightforward to show that the optimal solution to the KL-constrained reward maximization objective in Eq.”
Pretraining Language Models with Human Preferences (2023, methodology): “We experiment with four PHF objectives: conditional training (Keskar et al., 2019), dataset filtering, unlikelihood loss (Welleck et al., 2020) and two offline RL algorithms, reward-weighted regression (RWR; Peters & Schaal, 2007) and advantage-weighted regression (AWR; Peng et al., 2019).”
Benchmarking Deep Reinforcement Learning for Continuous Control (2016, methodology): “RWR: RWR is the only gradient-based algorithm we implemented that does not require any hyperparameter tuning.”
Aligning Language Models with Preferences through f-divergence Minimization (2023, background): “Other studies have also explored the use of forward KL divergence in RL (Peters & Schaal, 2007; Norouzi et al., 2016).”
Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “This objective differs from the ones used in the derivations of related algorithms, such as RWR and REPS (Peters & Schaal, 2007; Peters et al., 2010; Abdolmaleki et al., 2018), which maximize the expected return J ( π ) instead of the expected improvement.”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “Assuming r(τ) is positive and bounded and applying Jensen’s inequality results in a variety of algorithms (Peters and Schaal, 2007; Hachiya et al., 2009; Neumann, 2011; Abdolmaleki et al., 2018), including reward-weighted regression (Peters and Schaal, 2007), that all follow the following general re”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, background): “The analytic solution to Equation 1 can be obtained by enforcing the KKT conditions [13, 14, 12].”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “A monotonic positive reward transformation [24] also changes the reward distribution but is neither assured to keep optimal policies nor to have expected future rewards of zero.”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, background): “A number of works have leveraged this insight to develop RL algorithms [3, 29, 31, 32].”
Conditioning by adaptive sampling for robust design (2019, methodology): “(6) RWR —Reward Weighted Regression (Peters & Schaal, 2007), which is similar to DbAS , but without taking into account the oracle uncertainty.”
Learning robot in-hand manipulation with tactile features (2015, background): “These weighting factors need to be transformed to be strictly positive, as discussed in [39].”
Training Agents using Upside-Down Reinforcement Learning (2019, methodology): “More related is the work of Peters and Schaal [2007] who generalized the expectation maximization framework of Dayan and Hinton [1997] to convert policy search into (on-policy) weighted regression for the immediate reward setting.”
A Divergence Minimization Perspective on Imitation Learning Methods (2019, positive): “Building on such intuitions, a number of work extended RL algorithms based on picking another divergence metric, such as forward KL [25, 26], and demonstrated substantially improved empirical performances in certain situations.”
Simplifying Deep Reinforcement Learning via Self-Supervision (2021, methodology): “…(EM) Methods EM-based methods formulate reinforcement learning as a proba-0 bilistic inference problem and tackle it with Expectation-Maximization framework (Dayan and Hinton 1997; Peters and Schaal 2007; Kober and Peters 2009; Levine and Koltun 2013; Abdolmaleki et al. 2018).”
VIREL: A Variational Inference Framework for Reinforcement Learning (2018, methodology): “Inference algorithms such as EM can then be employed (Dayan & Hinton, 1997; Toussaint & Storkey, 2006; Peters & Schaal, 2007; Hachiya et al., 2009; Neumann, 2011; Abdolmaleki et al., 2018).”
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models (2023, methodology): “We show that ReST EM can be viewed as applying expectation-maximization for reinforcement learning (Dayan & Hinton, 1997; Peters & Schaal, 2007), which we present formally in Section 3.”