Good and safe uses of AI Oracles

By Stuart Armstrong. 2017

31 citations recorded by the source corpus.

It is possible that powerful and potentially dangerous artificial intelligence (AI) might be developed in the future. An Oracle is a design which aims to restrain the impact of a potentially dangerous AI by restricting the agent to no actions besides answering questions. Unfortunately, most Oracles will be motivated to gain more control over the world by manipulating users through the content of their answers, and Oracles of potentially high intelligence might be very successful at this \citep{DBLP:journals/corr/AlfonsecaCACAR16}. In this paper we present two designs for Oracles which, even under pessimistic assumptions, will not manipulate their users into releasing them and yet will still be incentivised to provide their users with helpful answers. The first design is the counterfactual Oracle -- which choses its answer as if it expected nobody to ever read it. The second design is the low-bandwidth Oracle -- which is limited by the quantity of information it can transmit.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Information Systems, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Information Systems

In Alignment & RLHF

Citations in the corpus

Alignment of Language Agents (2021, background): “As Armstrong et al. (2012) notes, however, there is still a potential risk that a sufficiently intelligent language agent could gain access to a less restricted action space, for example by manipulating its human gatekeepers to grant it physical actuators.”
Hidden Incentives for Auto-Induced Distributional Shift (2020, background): “…tampering with human-generated reward signals (Everitt & Hutter, 2018) (e.g. selecting news articles which manipulate user interests), and creating “self-fulﬁlling prophecies” (e.g. driving up the value of a held asset by publicly predicting its value will increase (Armstrong & O’Rorke, 2017)).”
Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective (2019, background): “ility solution. Recently there has been some work using neural networks to approximate counterfactual models (Buesing et al., 2019; Johansson et al., 2016). In the case of question-answering systems, Armstrong (2017) has suggested an elegant way to train these with a counterfactual objective by rand”