Artificial General Intelligence

By James Babcock, János Kramár, Roman V Yampolskiy. 2016

317 citations recorded by the source corpus.

There is considerable uncertainty about what properties, capabilities and motivations future AGIs will have. In some plausible scenarios, AGIs may pose security risks arising from accidents and defects. In order to mitigate these risks, prudent early AGI research teams will perform significant testing on their creations before use. Unfortunately, if an AGI has human-level or greater intelligence, testing itself may not be safe; some natural AGI goal systems create emergent incentives for AGIs to tamper with their test environments, make copies of themselves on the internet, or convince developers and operators to do dangerous things. In this paper, we survey the AGI containment problem – the question of how to build a container in which tests can be conducted safely and reliably, even on AGIs with unknown motivations and capabilities that could be dangerous. We identify requirements for AGI containers, available mechanisms, and weaknesses that need to be addressed.

Subfields: Information Systems and Management, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems and Management

In Signal Processing

Citations in the corpus

MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence (2017, background): “Introduction True human intelligence embraces social and collective wisdom, laying a foundation for general Artificial Intelligence (AI) (Goertzel and Pennachin 2007).”
Functional Decision Theory: A New Theory of Instrumental Rationality (2017, background): “Refer to Garber (1983), Gaifman (2004), Demski (2012), Hutter et al. (2013), Aaronson (2013), and Potyka and Thimm (2015) for a sample of discussion and research into inference in the face of uncertainty about logical facts.”
Logical Induction (2016, background): “Logical inductors are not the first computable procedure for assigning probabilities to sentences in a manner that is coherent in the limit; the algorithm of Demski (2012) also has this property.”
Artificial General Intelligence: Concept, State of the Art, and Future Prospects (2009, background): “My former colleague Shane Legg came up with “Artificial General Intelligence,” which Cassio and I liked, and adopted for the title of our edited book (Goertzel and Pennachin, 2007).”
Artificial Intelligence and Digital Pathology: Challenges and Opportunities (2018, unknown): “In contrast, with strong AI, also called artificial general intelligence (AGI), [24,25] we expect algorithms with human‑level intelligence, multitasking, and even consciousness as well as ethical cognition.”
Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Control (2020, background): “Multi-task Deep Reinforcement Learning Multi-task DRL has attracted research attention due to its potential for realizing AGI.”
Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement (2017, background): “2015), artificial general intelligence (Goertzel and Pennachin, 2007), universal artificial intelligence (Hutter, 2007), etc.”
Agent Incentives: A Causal Perspective (2021, methodology): “We call it an instrumental control incentive, as the control of X is a tool for achieving utility (cf. instrumental goals Omohundro 2008; Bostrom 2014).”
A Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy (2017, unknown): “179 Engagement on safety: unspecified Size: medium-small 174 http://kimera.ai 175 http://kimera.ai 176 Boyle (2016); Jee (2016) 177 Boyle (2016) 178 http://kimera.ai/nigel 179 http://kimera.ai/company”
Safely Interruptible Agents (2016, background): “…designing reward functions can be tricky [Humphrys, 1996, Murphy, 2013]; the agent may find unpredictable and undesirable shortcuts to receive rewards, and the reward function needs to be adjusted in accordance—the problem can go as far as to nullify any reward function [Ring and Orseau, 2011].”
AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors in Agents (2023, unknown): “These advancements edge us closer to realizing the concept of artificial general intelligence (AGI) (Goertzel & Pennachin, 2007; Clune, 2019) that can generalize across a broader range of tasks.”
The Ethics of Artificial Intelligence (2014, unknown): “While this subfield of Artificial Intelligence is only just coalescing, “Artificial General Intelligence” (hereafter, AGI) is the emerging term of art used to denote “real” AI (see, e.g., the edited volume Goertzel and Pennachin 2006).”
Multiagent Bidirectionally-Coordinated Nets for Learning to Play StarCraft Combat Games (2017, background): “Yet, true human intelligence embraces social and collective wisdom [9], which lays an essential foundation for reaching the grand goal of Artificial General Intelligence (AGI) [10].”
On the Machine Learning of Ethical Judgments from Natural Language (2022, background): “At its core, this vision is one of artificial general intelligence (Goertzel and Pennachin, 2007), and similar in scope and intent to the Moral Machine exper-iment (Awad et al., 2018), which also sought to leverage the “wisdom of the crowd” in proposing frameworks for how a future self-driving car…”
Reward is enough (2021, background): “General intelligence is sometimes formalised by a set of environments that measures the agent’s capabilities across a variety of different goals and contexts [25,14].”