The Cross Entropy Method for Fast Policy Search

By Shie Mannor, Reuven Y. Rubinstein, Yohai Gat. 2003

207 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Learning Parameterized Skills (2012, methodology): “Direct policy search can be performed in three main ways: (1) by using black-box local search methods, such as hill climbing or the NelderMead simplex technique [34]; (2) by using gradient descent methods [86, 6, 72, 9]; or (3) by using EM or crossentropy approaches [31, 46].”
Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks (2016, methodology): “Compared to algorithms for eﬃcient direct exploration in parameter space (see e.g., [18, 27, 37, 25]), the novelty of our algorithm is to mix exploration through parameter randomization and plain gradient descent.”
Path Integral Policy Improvement with Covariance Matrix Adaptation (2012, methodology): “Before comparing the algorithms, we ﬁrst present the evaluation task used in the paper.”
V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control (2019, methodology): “This is similar to the technique used in the Cross Entropy Method (CEM) (Mannor et al., 2003) and Covariance Matrix Adaptation - Evolutionary Strategy (CMAES) (Hansen et al.”
Stein Variational Policy Gradient (2017, methodology): “Instead of computing the ﬁnite difference for each individual parameter, rollout efﬁcient methods such as SPSA (Spall, 1998), PEPG (Sehnke et al., 2010) and evolutionary strategy approximations (Hansen, 2006, Mannor et al., 2003) have been proposed.”
An Online Learning Approach to Model Predictive Control (2019, background): “CEM has found applicability in reinforcement learning [19, 21, 26], motion planning [17, 18], and MPC [9, 32].”
CAQL: Continuous Action Q-Learning (2019, background): “By contrast, in problems with a continuous action spaces (e.g., robotics (Peters & Schaal, 2006)), policy-based algorithms, such as policy gradient (PG) (Sutton et al., 2000; Silver et al., 2014) or cross-entropy policy search (CEPS) (Mannor et al., 2003; Kalashnikov et al., 2018), which directly le”
A Tutorial on the Cross-Entropy Method (2005, background): “Extensions, other reward criteria, and further applications appear in (Mannor et al., 2003).”
Deriving and improving CMA-ES with information geometric trust regions (2017, methodology): “Subsequently, a new search distribution is computed by either computing gradient based updates [19], expectation-maximisationbased updates [7, 12], evolutionary strategies [10], the cross-entropy method [14] or information-theoretic policy updates [1], such that the individuals with higher fitness w”
Simplifying Deep Reinforcement Learning via Self-Supervision (2021, methodology): “…and episode-level reward, but treats the problem as a black-box optimization and directly search the policy in parameter space, such as Cross-Entropy Method (CEM) (Mannor, Rubinstein, and Gat 2003), random search (Mania, Guy, and Recht 2018), and evolution strategies (Salimans et al. 2017).”
Learning To Reach Goals Without Reinforcement Learning (2019, methodology): “The cross-entropy method (Mannor et al., 2003), self-imitation learning (Oh et al.”
Contrasting Exploration in Parameter and Action Space: A Zeroth-Order Optimization Perspective (2019, methodology): “…by employing , e.g., ﬁnite-diﬀerence-like methods to compute estimates of the gradient with respect to policy parameters (Bagnell and Schneider, 2001; Mannor et al., 2003; Heidrich-Meisner and Igel, 2008; Tesch et al., 2011; Sehnke et al., 2010; Salimans et al., 2017; Mania et al., 2018).”
Learning to Reach Goals via Iterated Supervised Learning (2019, methodology): “Direct policy search methods [Mannor et al., 2003, Peters and Schaal, 2007, Theodorou et al., 2010, Goschin et al., 2013, Norouzi et al., 2016, Nachum et al., 2016] selectively weight policies or trajectories by their performance during learning, as measured by the environment’s reward function or…”
Variational Inference for Policy Search in changing situations (2011, methodology): “Many policy search algorithm like the CMA-ES (HeidrichMeisner & Igel, 2009), Cross-Entropy search (Mannor et al., 2003) or the PoWER (Kober & Peters, 2009) algorithm are limited to the single-situation setting.”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “Actor-only policy search also includes gradient-free optimization methods [81, 257, 142, 80, 384], which are usable even when the policy is non-differentiable, and in some cases (e.”