Engineering a Safer World: Systems Thinking Applied to Safety

By Nancy G. Leveson. 2012

A new approach to safety, based on systems thinking, that is more effective, less costly, and easier to use than current techniques.Engineering has experienced a technological revolution, but the basic engineering techniques applied in safety and reliability engineering, created in a simpler, analog world, have changed very little over the years. In this groundbreaking book, Nancy Leveson proposes a new approach to safety?more suited to today's complex, sociotechnical, software-intensive world?based on modern systems thinking and systems theory. Revisiting and updating ideas pioneered by 1950s aerospace engineers in their System Safety concept, and testing her new model extensively on real-world examples, Leveson has created a new approach to safety that is more effective, less expensive, and easier to use than current techniques.Arguing that traditional models of causality are inadequate, Leveson presents a new, extended model of causation (Systems-Theoretic Accident Model and Processes, or STAMP), then shows how the new model can be used to create techniques for system safety engineering, including accident analysis, hazard analysis, system design, safety in operations, and management of safety-critical systems. She applies the new techniques to real-world events including the friendly-fire loss of a U.S. Blackhawk helicopter in the first Gulf War; the Vioxx recall; the U.S. Navy SUBSAFE program; and the bacterial contamination of a public water supply in a Canadian town. Leveson's approach is relevant even beyond safety engineering, offering techniques for ?reengineering? any large sociotechnical system to improve safety and manage risk.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Hazard Analysis Framework for Code Synthesis Large Language Models (2022, neutral): “There are numerous approaches, techniques, and levels of rigor in carrying out a hazard analysis and a risk assessment, and we refer to existing literature for further detail [26].”
Unsolved Problems in ML Safety (2021, background): “In a report for the Department of Defense, Frola and Miller [55] observe that approximately 75% of the most critical decisions that determine a system’s safety occur early in development [121].”
Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing (2020, methodology): “For example, a power plant can be consistently productive while causing harm to the environment through pollution [42].”
A survey of approaches combining safety and security for industrial control systems (2015, background): “System theoretic accident model and processes (STAMP) is an accident causality model developed by Leveson [104] that accounts for new causal factors associated with software, human decision making, new technology, social and organizational design, and increasing complexity.”
Bridging the Gap Between Ethics and Practice (2020, background): “Leveson [68] develops a systems engineering approach to safety engineering that includes design, hazard analysis, and failure investigations.”
Developing a Scale to Measure Tourist Perceived Safety (2021, background): “Safety system theory identifies the common patterns of accidents and safety management from the perspective of a system, and its main research concepts include accident and safety systems (Leveson 2011; Lower, Magott, and Skorupski 2018).”
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction (2022, background): “Harm to social systems is thus about how algorithmic systems adversely shape the emergent properties [129] of social systems [162].”
Improving Alignment and Robustness with Circuit Breakers (2024, unknown): “…we make models intrinsically safer and reduce their risks by removing intrinsic model hazards —their ability to produce harmful outputs—rather than removing specific vulnerabilities with adversarial training, and rather than attempting to reduce exposure to attacks with input filters [28, 21].”
Systems-Theoretic Safety Assessment of Robotic Telesurgical Systems (2015, methodology): “…was not performed, ii) a control action was performed in a wrong state, leading to a hazard, iii) a control action was performed at an incorrect time, iv) a control action was performed for an incorrect duration, v) a control action was provided, but not followed by the controlled process [4].”
Sociotechnical Harms: Scoping a Taxonomy for Harm Reduction (2022, background): “Harm to social systems is thus about how algorithmic systems adversely shape the emergent properties [110] of social systems [144].”
The bureaucratization of safety (2014, background): “…outcomes and risk management (Hale et al., 2013), and a differentiation of what works for safety in certain application areas (Grote, 2012) or at existing levels of safety (Amalberti, 2001), or between process safety, system safety and personal safety (Grote and Weichbrodt, 2013; Leveson, 2012).”
An Overview of Catastrophic AI Risks (2023, background): “By embedding these principles in their approach to safety, AI labs can enhance the overall security and resilience of their AI systems [94, 102].”
Towards the assessment of potential impact of unmanned vessels on maritime transportation safety (2017, background): “This highlights the importance of applying a systemic approach to the design of unmanned ship and the operational environment she is anticipated to work within [52].”