Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy

By Brian D. Ziebart. 2010

770 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “ the correspondingmodels. Less explored in the recent literature is the connection between maximum entropy reinforcement learning and robust control. Although some work has hinted at this connection (Ziebart, 2010), the potential for maximum entropy reinforcement learning to produce policies that ar”
Imitating Interactive Intelligence (2020, methodology): “Generative Adversarial Imitation Learning (GAIL) (Ho and Ermon, 2016) is an algorithm closely related to IRL (Ziebart, 2010; Finn et al., 2016).”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, background): “More importantly, the maximum entropy formulation provides a substantial improvement in exploration and robustness: as discussed by Ziebart (2010), maximum entropy policies are robust in the face of model and estimation errors, and as demonstrated by (Haarnoja et al., 2017), they improve…”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “In this paper, we study the optimal state and action values for a softmax form of temporal consistency [48, 47, 7], which arises by augmenting the standard expected reward objective with a discounted entropy regularizer.”
Dual Policy Iteration (2018, methodology): “10 for η exactly by LQR [39].”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, neutral): “rcement learning, which adds an entropy regularization `bonus' to prevent early convergence to deterministic policies and thereby encourages exploration, though there are no guarantees on regret [112, 113, 54, 63, 2]. In our work the soft-max operator and entropy regularization arise naturally ”
What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020, background): “We consider a number of different policy regularizers both for penalty (choice C33) and constraint regularization (choice C34): • Entropy H(π(·|s)) — it encourages the policy to try diverse actions [4].”
End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, methodology): “a) Maximum Entropy RL: The particular reinforcement learning framework that we will use, called maximum entropy RL [46, 14, 15], also maximizes the entropy of the resulting distribution, resulting in the modified objective”
Soft Actor-Critic Algorithms and Applications (2018, background): “(Ziebart, 2010) generalizes the standard objective by augmenting it with an entropy term, such that the optimal policy additionally aims to maximize its entropy at each visited state:”
Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization (2016, methodology): “babilistic models that explain suboptimal behavior as noise (Ramachandran & Amir, 2007; Ziebart et al., 2008). We take the latter approach in this work, building on the maximum entropy IOC model (Ziebart, 2010). Although the probabilistic model mitigates some of the difﬁculties with IOC, there i”
An Algorithmic Perspective on Imitation Learning (2018, methodology): “ear reward [Boularias et al., 2011, Kalakrishnan et al., 2013] [Abbeel and Ng, 2004, Ratliﬀ et al., 2006b, Silver et al., 2010, Ramachandran and Amir, 2007, Choi and Kim, 2011b, Ziebart et al., 2008, Ziebart, 2010, Levine and Koltun, 2012, Hadﬁeld-Menell et al., 2016] Nonlinear reward [Finn et al., ”
Disentangling the independently controllable factors of variation by interacting with the world (2018, positive): “Thus, our total objective along entire trajectories is a lower bound on the causal [Ziebart, 2010] or directed [Massey, 1990] information which is a measure of the causality the process φ exercises on the process h .”
Learning Robust Rewards with Adversarial Inverse Reinforcement Learning (2017, methodology): “ion 5. 3 BACKGROUND Our inverse reinforcement learning method builds on the maximum entropy IRL framework (Ziebart et al., 2008), which considers an entropy-regularized Markov decision process (MDP) (Ziebart, 2010), deﬁned by the tuple (S;A;T;R; ;ˆ 0). S;Aare the state and action spaces, respectivel”
Batch Policy Learning under Constraints (2019, background): “• Entropy regularized RL (Haarnoja et al., 2017; Ziebart, 2010) maps to policy-dependent constraint cost g(x) = H ( π(·|x) ) , where H measures conditional entropy.”