Reward-rational (implicit) choice: A unifying formalism for reward learning

By Hong Jun Jeon, Smitha Milli, Anca Dragan. 2020

208 citations recorded by the source corpus.

It is often difficult to hand-specify what the correct reward function is for a task, so researchers have instead aimed to learn reward functions from human behavior or feedback. The types of behavior interpreted as evidence of the reward function have expanded greatly in recent years. We've gone from demonstrations, to comparisons, to reading into the information leaked when the human is pushing the robot away or turning it off. And surely, there is more to come. How will a robot make sense of all these diverse types of behavior? Our key insight is that different types of behavior can be interpreted in a single unifying formalism - as a reward-rational choice that the human is making, often implicitly. The formalism offers both a unifying lens with which to view past work, as well as a recipe for interpreting new sources of information that are yet to be uncovered. We provide two examples to showcase this: interpreting a new feedback type, and reading into how the choice of feedback itself leaks information about the reward.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Molecular Biology, Alignment & RLHF, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Sergey Levine
  4. Percy Liang
  5. Kyunghyun Cho
  6. Yoshua Bengio
  7. Luke S. Zettlemoyer
  8. Yejin Choi

In Computational Theory and Mathematics

  1. Anca Dragan
  2. Dylan Hadfield-Menell
  3. Daniel Klein
  4. Jessy Lin
  5. Daniel Fried
  6. Fabio Cuzzolin
  7. Stewart Slocum
  8. Christelle Langley

In Molecular Biology

  1. Stewart Slocum
  2. Peter Hase
  3. Barbara J. Sahakian
  4. Karthik Narasimhan
  5. David Krueger
  6. Dorsa Sadigh
  7. T. Griffiths
  8. Weizhu Chen

In Alignment & RLHF

  1. Angelica Chen
  2. J'er'emy Scheurer
  3. Tomasz Korbak
  4. Jeff Wu
  5. Anca Dragan
  6. Ryan J. Lowe
  7. Long Ouyang
  8. Daniel M. Ziegler

In Reinforcement Learning

  1. Anca Dragan
  2. Sergey Levine
  3. Dorsa Sadigh
  4. Daniel Klein
  5. Dylan Hadfield-Menell
  6. Daniel Fried
  7. Theodore R. Sumers
  8. Karthik Narasimhan

Citations in the corpus