Thinking Inside the Box: Controlling and Using an Oracle AI

By Stuart Armstrong, Anders Sandberg, Nick Bostrom. 2012

121 citations recorded by the source corpus.

Subfields: Safety Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety Research

In Safety, Robustness & Interpretability

Citations in the corpus

GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “Because pretrained LLMs are not directly optimized for the factual accuracy of their predictions, it is possible they avoid some of the traditional problems with tool or oracle AI (Armstrong et al., 2012), such as the incentive to produce manipulative answers (Demski, 2019).”
Alignment of Language Agents (2021, background): “While some work has considered the containment of Oracle AI (Armstrong et al., 2012), which we discuss in Section 2, behavioral issues with language agents have received comparatively little attention compared to the delegate agent case.”
Artificial Intelligence, Values, and Alignment (2020, background): “One important distinction is between ‘oracle AI’, which ‘does not act in the world except by answering questions’ (Armstrong et al., 2012), and forms of AI that have a wider range of capabilities.”
Good and safe uses of AI Oracles (2017, background): “An Oracle is a common high-level idea for ensuring the safe use of powerful Ar-tiﬁcial Intelligences (AIs) [Armstrong et al., 2012, Yampolskiy, 2012, Bostrom, 2014].”
AGI Safety Literature Review (2018, background): “Armstrong (2017a) and Armstrong, Sandberg, et al. (2012) argue that oracles that only answer questions may be more safe than other types of AGI.”
Understanding Agent Incentives using Causal Influence Diagrams. Part I: Single Action Settings (2019, background): “t continuously interact with the world to achieve some long-term goal. Another possibility is to construct a pure questionanswering system (QA-system), with the only goal to correctly answer queries (Armstrong et al., 2012). QA-Systems have some safety beneﬁts, as they only aﬀect the world through t”
Formalizing Convergent Instrumental Goals (2016, background): “Some research of this form is already under way, under the name of “limited optimization” or “domesticity”; see the works of Armstrong et al. (2012), Taylor (2015), and others.”
When Combating Hype, Proceed with Caution (2021, background): “Systems that interact with humans only through text, or systems whose goals are circumscribed to a well-deﬁned task like question answering, are not exempt from this concern (Armstrong et al., 2012).”
Quantilizers: A Safer Alternative to Maximizers for Limited Optimization (2016, background): “A grant from the Future of Life Institute helped fund this work.”
Motivated Value Selection for Artificial Agents (2015, background): “25It could be set to zero, but this might interfere with the AI’s computation of counterfactuals.”
Aligning Superintelligence with Human Interests: A Technical Research Agenda (2015, background): “For a discussion, see Armstrong, Sandberg, and Bostrom (2012).”
Low Impact Artificial Intelligences (2017, background): “Unlike other boxing suggestions [ASB12, Bos14, Yam12], the aim here is not to prevent the AI from having an impact by conﬁning it.”
Artificial Intelligence and Marketing: Pitfalls and Opportunities (2020, background): “At the very early design stages, marketing organizations need to put in place mechanisms to control, stop, and override AI algorithms in real-time (Armstrong, Sandberg, & Borstrom, 2012; Russell 2019).”