Speculations Concerning the First Ultraintelligent Machine

By Irving John Good. 1965

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Philosophy

Citations in the corpus

When Will AI Exceed Human Performance? Evidence from AI Experts (2017, background): “Some authors have argued that once HLMI is achieved, AI systems will quickly become vastly superior to humans in all tasks [3, 13].”
Research Priorities for Robust and Beneficial Artificial Intelligence (2015, background): “A related verification research topic that is distinctive to long-term concerns is the verifiability of systems that modify, extend, or improve themselves, possibly many times in succession (Good 1965; Vinge 1993).”
Scruples: A Corpus of Community Ethical Judgments on 32, 000 Real-Life Anecdotes (2020, background): “Early on, I.J. Good raised the possibility of an “intelligence explosion” and the great beneﬁts, as well as dangers, such an event may pose (Good 1966).”
Learning What to Value (2011, background): “Worse, increasing intelligence could trigger an intelligence explosion [1, 8] in which AI-RL repeatedly self-improves until it far surpasses our ability to contain it.”
The Role of Cooperation in Responsible AI Development (2019, background): “7 The possibility of discontinuous progress in AI is discussed by Good (1966), Chalmers (2009), Yudkowsky (2013), Shanahan (2015) and Bostrom (2017b).”
Thinking Inside the Box: Controlling and Using an Oracle AI (2012, background): “Another avenue is to start with a ‘seed’ AI, of limited intelligence but with the ability to self-improve, in the hope that it will transform itself into a much more intelligent entity, and continue doing so again and again using its growing intelligence (Good 1965; Yudkowsky 2001c).”
The Ethics of Artificial Intelligence (2014, unknown): “Superintelligence I. J. Good (1965) set forth the classic hypothesis concerning superintelligence: that an AI sufficiently intelligent to understand its own design could redesign itself or create a successor system, more intelligent, which could then redesign itself yet again to become even more…”
Artificial Intelligence as a Positive and Negative Factor in Global Risk (2006, background): “One often encounters skepticism about this scenario - what Good (1965) called an "intelligence explosion" - because progress in Artificial Intelligence has the reputation of being very slow.”
Measuring universal intelligence: Towards an anytime intelligence test (2010, background): “A more explicit link to artificial intelligence is made through the term “intelligence explosion”, introduced by Good [59] in 1965.”
AGI Safety Literature Review (2018, background): “If the pace of this process increases, we may see an intelligence explosion once a critical level of self-improvement capability has been reached (Bostrom, 2014; Good, 1966; Hutter, 2012a; Kurzweil, 2005; Vinge, 1993; Yud-kowsky, 2008b).”
The Nature of Self-Improving Artificial Intelligence (2008, background): “Irving Good [1] envisioned the consequences in 1965:”
Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective (2019, background): “eward function with an always maximized version. (e) (Hypothetical) It has been hypothesized that suﬃciently intelligent AI systems will try to further improve their cognitive powers (Chalmers, 2010; Good, 1966; Omohundro, 2008). Such a self-improvement may involve substantial modiﬁcations of the ag”
A Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy (2017, methodology): “The term ultraintelligence has also been used in this context (Good 1965) but is less common.”
Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda (2017, background): “At the same time, unpredictable behavior from smarter-than-human systems could cause catastrophic damage, if they are not aligned with human interests (Yudkowsky 2008).”