Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic

By Mihály Bárász, Paul Christiano, Benja Fallenstein, Marcello Herreshoff, Patrick LaVictoire, Eliezer Yudkowsky. 2014

30 citations recorded by the source corpus.

We consider the one-shot Prisoner's Dilemma between algorithms with read-access to one anothers' source codes, and we use the modal logic of provability to build agents that can achieve mutual cooperation in a manner that is robust, in that cooperation does not require exact equality of the agents' source code, and unexploitable, meaning that such an agent never cooperates when its opponent defects. We construct a general framework for such "modal agents", and study their properties.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “Yet while there has been some of progress on this research agenda (Barasz et al., 2014; Leike et al., 2016; Garrabrant et al., 2016; Everitt, 2018), some questions turned out to be quite difﬁcult.”
Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda (2017, background): “They have found that it is possible for some agents to achieve robust cooperation in the one-shot prisoner’s dilemma while remaining unexploitable (Bárász et al. 2014).”
Problem Class Dominance in Predictive Dilemmas (2014, background): “Such a situation has been analyzed using modal agents in the context of a one shot prisoners’ dilemma, and the agent which has so far enjoyed the greatest success was inspired by Timeless Decision Theory (Barasz et al, 2014).”
Aligning Superintelligence with Human Interests: A Technical Research Agenda (2015, background): “Recently, B´ar´asz et al. (2014) developed a concrete model, together with a Haskell implementation, of multi-agent games where agents have access to each others’ source code and base their decisions on what they can prove about their opponent.”