Reward-rational (implicit) choice: A unifying formalism for reward learning
By Hong Jun Jeon, Smitha Milli, Anca Dragan. 2020
208 citations recorded by the source corpus.
It is often difficult to hand-specify what the correct reward function is for a task, so researchers have instead aimed to learn reward functions from human behavior or feedback. The types of behavior interpreted as evidence of the reward function have expanded greatly in recent years. We've gone from demonstrations, to comparisons, to reading into the information leaked when the human is pushing the robot away or turning it off. And surely, there is more to come. How will a robot make sense of all these diverse types of behavior? Our key insight is that different types of behavior can be interpreted in a single unifying formalism - as a reward-rational choice that the human is making, often implicitly. The formalism offers both a unifying lens with which to view past work, as well as a recipe for interpreting new sources of information that are yet to be uncovered. We provide two examples to showcase this: interpreting a new feedback type, and reading into how the choice of feedback itself leaks information about the reward.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Molecular Biology, Alignment & RLHF, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Sergey Levine
- Percy Liang
- Kyunghyun Cho
- Yoshua Bengio
- Luke S. Zettlemoyer
- Yejin Choi
In Computational Theory and Mathematics
- Anca Dragan
- Dylan Hadfield-Menell
- Daniel Klein
- Jessy Lin
- Daniel Fried
- Fabio Cuzzolin
- Stewart Slocum
- Christelle Langley
In Molecular Biology
- Stewart Slocum
- Peter Hase
- Barbara J. Sahakian
- Karthik Narasimhan
- David Krueger
- Dorsa Sadigh
- T. Griffiths
- Weizhu Chen
In Alignment & RLHF
- Angelica Chen
- J'er'emy Scheurer
- Tomasz Korbak
- Jeff Wu
- Anca Dragan
- Ryan J. Lowe
- Long Ouyang
- Daniel M. Ziegler
In Reinforcement Learning
- Anca Dragan
- Sergey Levine
- Dorsa Sadigh
- Daniel Klein
- Dylan Hadfield-Menell
- Daniel Fried
- Theodore R. Sumers
- Karthik Narasimhan
Citations in the corpus
- Learning to summarize from human feedback (2020, background): “There is also a rich landscape of human feedback methods beyond binary comparisons that could be explored for training models [28, 17, 44, 64].”
- Learning Rewards from Linguistic Feedback (2020, background): “This idea has been extended to settings in which agents are provided with intentionally informative demonstrations (Ho et al. 2016), a variety of human acts (Jeon, Milli, and Dragan 2020), or themselves act informatively (Dragan, Lee, and Srinivasa 2013; Hadfield-Menell et al. 2016).”
- Inferring Rewards from Language in Context (2022, methodology): “…communities and has not incorporated language, rather using techniques such as inverse reinforcement learning (IRL; Ng and Rus-sell 2000; Ratliff et al. 2006; Ziebart et al. 2008; Hadfield-Menell et al. 2017; Jeon et al. 2020) to infer the rewards that underlie human demonstrations of actions.”
- Theory of Mind and Preference Learning at the Interface of Cognitive Science, Neuroscience, and AI: A Review (2022, background): “…shows that there has been significant progress on the theoretical side, especially in preference ToM, e.g., combining different information sources about human preferences (Hong et al., 2020) and integrating uncertainty (Hadfield-Menell et al., 2017b) and human interaction (Shah et al., 2021a).”
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, methodology): “Skalse and Abate [2022a] formally study the effect of inverse reinforcement learning with a misspecified Boltzmann model, which is also common [Jeon et al., 2020].”
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “Traditional approaches to mitigate these limitations typically employ additional training, involving behavior cloning, reinforcement learning, and self-training (Saunders et al., 2022; Stiennon et al., 2020; Jeon et al., 2020; Bai et al., 2022b).”