Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims

By Miles Brundage, S. Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian K. Hadfield, Heidy Khlaaf, Jingying Yang, H. Toner, Ruth C. Fong and 49 more. 2020

480 citations recorded by the source corpus.

With the recent wave of progress in artificial intelligence (AI) has come a growing awareness of the large-scale impacts of AI systems, and recognition that existing regulations and norms in industry and academia are insufficient to ensure responsible AI development. In order for AI developers to earn trust from system users, customers, civil society, governments, and other stakeholders that they are building AI responsibly, they will need to make verifiable claims to which they can be held accountable. Those outside of a given organization also need effective means of scrutinizing such claims. This report suggests various steps that different stakeholders can take to improve the verifiability of claims made about AI systems and their associated development processes, with a focus on providing evidence about the safety, security, fairness, and privacy protection of AI systems. We analyze ten mechanisms for this purpose--spanning institutions, software, and hardware--and make recommendations aimed at implementing, exploring, or improving those mechanisms.

Subfields: Artificial Intelligence, Safety Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Safety, Robustness & Interpretability

Citations in the corpus

Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, methodology): “One potentially useful tool for addressing harm is red teaming—using manual or automated methods to adversarially probe a language model for harmful outputs, and then updating the model to avoid such outputs [42, 20, 3, 11].”
Predictability and Surprise in Large Generative Models (2022, background): “We can model this on the ‘red team’ approach which is popular in the computer security industry and can be applied in an AI context [6, 13].”
The Gradient of Generative AI Release: Methods and Considerations (2023, background): “Bounty programs, from bug bounties to bias bounties [82], can raise unforeseen safety issues and strengthen trust in a system [18].”
A Hazard Analysis Framework for Code Synthesis Large Language Models (2022, background): “Additionally, Codex synthesizes Python, Javascript, Typescript, and Ruby code, all of which are not amenable to verification [11]; thus it would be difficult to leverage formal specification and verification techniques to evaluate the generated output.”
Unsolved Problems in ML Safety (2021, background): “Beyond technical work, policy and governance work will be integral to safe deployment [42, 11, 15, 210, 24].”
Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “It is universally agreed that interpretability is a key element of trust for AI models (Wagstaff, 2012; Rudin and Wagstaff, 2014; Lo Piano, 2020; Ashoori and Weisz, 2019; Thiebes et al., 2020; Spiegelhalter, 2020; Brundage et al., 2020).”
Informed Machine Learning – A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems (2019, background): “Another important aspect is that a purely data-driven model might not meet constraints such as dictated by natural laws, or given through regulatory or security guidelines, which are important for trustworthy AI [8].”
The alignment problem from a deep learning perspective (2022, background): “Such cooperation would become more viable given mechanisms for allowing AI developers to certify properties of training runs without leaking information about the code or data they used [Brundage et al., 2020].”
The ethics of algorithms: key problems and solutions (2020, unknown): “This includes information ranging from characteristics like geographical location and age to information on specific behaviour and preferences, including what type of content a person is likely to seek the most on a given platform (Chakraborty et al. 2019).”
Secure, privacy-preserving and federated machine learning in medical imaging (2020, background): “(8) The development of auditable and objectively trustworthy systems 99 (that is, not relying on subjective assertions—for example, by governments) will promote the universal acceptance of secure and private AI solutions by individuals and policymakers.”
OpenAI o1 System Card (2024, unknown): “We also build on external red teaming practices learnt over prior models [10, 11, 12, 13] and take advantage of the increasingly mature ecosystem for red teaming.”
Harms from Increasingly Agentic Algorithmic Systems (2023, methodology): “The likely failures of a more agentic system (relative to what has been deployed already) are certainly a barrier to deployment – given the disproportionate impact of these failures on already marginalized groups, we would hope and advocate for restrictions on deployment [Brundage et al. 2020].”
Trustworthy AI: From Principles to Practices (2021, background): “• We consider ethical requirements with broad concerns in recent literature [9, 57, 121, 145, 178, 199, 218, 301, 334], including fairness, privacy, and accountability.”