Score-based Inverse Reinforcement Learning

By Layla El Asri, Bilal Piot, Matthieu Geist, Romain Laroche, Olivier Pietquin. 2016

41 citations recorded by the source corpus.

This paper reports theoretical and empirical results obtained for the score-based Inverse Reinforcement Learning (IRL) algorithm. It relies on a non-standard setting for IRL consisting of learning a reward from a set of globally scored trajectories. This allows using any type of policy (optimal or not) to generate trajectories without prior knowledge during data collection. This way, any existing database (like logs of systems in use) can be scored a posteriori by an expert and used to learn a reward function. Thanks to this reward function, it is shown that a near-optimal policy can be computed. Being related to least-square regression, the algorithm (called SBIRL) comes with theoretical guarantees that are proven in this paper. SBIRL is compared to standard IRL algorithms on synthetic data showing that annotations do help under conditions on the quality of the trajectories. It is also shown to be suitable for real-world applications such as the optimisation of a spoken dialogue system.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning from Human Preferences (2017, background): “…reinforcement learning from human ratings or rankings, including Akrour et al. (2011), Pilarski et al. (2011), Akrour et al. (2012), Wilson et al. (2012), Sugiyama et al. (2012), Wirth and Fürnkranz (2013), Daniel et al. (2015), El Asri et al. (2016), Wang et al. (2016), and Wirth et al. (2016).”
Scalable agent alignment via reward modeling: a research direction (2018, background): “…of the desired behavior (Russell, 1998; Ng & Russell, 2000; Abbeel & Ng, 2004; Argall et al., 2009); providing feedback in the form of scores (El Asri et al., 2016), actions (Grifﬁth et al., 2013), value (Knox & Stone, 2009), advantage (MacGlashan et al., 2017), or preferences over trajectories…”
Reward learning from human preferences and demonstrations in Atari (2018, background): “…learning from human ratings or rankings (Wirth et al., 2017): Knox and Stone (2009), Pilarski et al. (2011), Akrour et al. (2012), Wilson et al. (2012), Wirth and Fürnkranz (2013), Daniel et al. (2015), El Asri et al. (2016), Wirth et al. (2016), Mathewson and Pilarski (2017), and others.”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, methodology): “Other methods (Burchﬁel et al., 2016; El Asri et al., 2016) have proposed the use of quantitatively scored trajectories as opposed to qualitative pairwise preferences over demonstrations.”
Explore, Exploit or Listen: Combining Human Feedback and Policy Model to Speed up Deep Reinforcement Learning in 3D Worlds (2017, background): “Inverse reinforcement learning (Abbeel and Ng 2004; El Asri et al. 2016), on the other hand, seeks to directly engineer a reward function based on examples of optimal behavior provided by human trainers.”