Embedded Agency

By Abram Demski, Scott Garrabrant. 2019

38 citations recorded by the source corpus.

Traditional models of rational action treat the agent as though it is cleanly separated from its environment, and can act on that environment from the outside. Such agents have a known functional relationship with their environment, can model their environment in every detail, and do not need to reason about themselves or their internal parts. We provide an informal survey of obstacles to formalizing good reasoning for agents embedded in their environment. Such agents must optimize an environment that is not of type "function"; they must rely on models that fit within the modeled environment; and they must reason about themselves as just another physical system, made of parts that can be modified and that can work at cross purposes.

Subfields: Computational Mechanics, History and Philosophy of Science, Organizational Behavior and Human Resource Management, Safety, Robustness & Interpretability, Philosophy

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Ilya Sutskever
2. Pavel Izmailov
3. Collin Burns

In History and Philosophy of Science

1. Pavel Izmailov

In Safety, Robustness & Interpretability

1. Tom Everitt
2. Shane Legg
3. Miljan Martic
4. Jan Leike
5. David Krueger
6. Vishal Maini
7. Ilya Sutskever
8. Leo Gao

In Philosophy

1. David Krueger
2. Tom Everitt
3. Leo Gao
4. Miljan Martic
5. Jan Leike
6. Shane Legg
7. Pavel Izmailov
8. Collin Burns

Citations in the corpus

• Scalable agent alignment via reward modeling: a research direction (2018, background): “In particular, many of their problems aim to address perceived difficulties in applying current notions of optimal behavior to agents which are part of their environment (Orseau & Ring, 2012) and thus may not remain cleanly delineated from it (Demski & Garrabrant, 2018).”
• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, background): “…head-on—but been restricted to primarily theoretical frameworks and toy problems (Irving et al., 2018; Chris-tiano et al., 2018; Leike et al., 2018; Demski & Garrabrant, 2019; Hubinger et al., 2019), or empirically studied humans supervising today’s models—without addressing the core challenges…”