Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning

By Aviv Ovadya, Jess Whittlestone. 2019

31 citations recorded by the source corpus.

The aim of this paper is to facilitate nuanced discussion around research norms and practices to mitigate the harmful impacts of advances in machine learning (ML). We focus particularly on the use of ML to create "synthetic media" (e.g. to generate or manipulate audio, video, images, and text), and the question of what publication and release processes around such research might look like, though many of the considerations discussed will apply to ML research more broadly. We are not arguing for any specific approach on when or how research should be distributed, but instead try to lay out some useful tools, analogies, and options for thinking about these issues. We begin with some background on the idea that ML research might be misused in harmful ways, and why advances in synthetic media, in particular, are raising concerns. We then outline in more detail some of the different paths to harm from ML research, before reviewing research risk mitigation strategies in other fields and identifying components that seem most worth emulating in the ML and synthetic media research communities. Next, we outline some important dimensions of disagreement on these issues which risk polarizing conversations. Finally, we conclude with recommendations, suggesting that the machine learning community might benefit from: working with subject matter experts to increase understanding of the risk landscape and possible mitigation strategies; building a community and norms around understanding the impacts of ML research, e.g. through regular workshops at major conferences; and establishing institutions and systems to support release practices that would otherwise be onerous and error-prone.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Release Strategies and the Social Impacts of Language Models (2019, background): “We identify areas to improve in legal and technical infrastructure for model sharing below [51]: • Scalability: Currently, agreements require fine-detail discussion and negotiation.”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “Within the broader context of value-sensitive design, and absent responsible release norms in the field (Ovadya and Whittlestone, 2019; Liang et al., 2022), we propose the following elements of a framework to aid researchers in deliberating safer release, and guidance to support learning during and ”
The Gradient of Generative AI Release: Methods and Considerations (2023, background): “While limiting access can prevent some malicious uses and is often a suggested action to minimize misuse [71], systems can still be vulnerable to attacks with only querying functionality available [21].”
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims (2020, background): “Incident sharing is closely related to but distinct from responsible publication practices in AI and coordinated disclosure of cybersecurity vulnerabilities [55].”
The tension between openness and prudence in AI research (2019, background): “However, concerns about potential harms arising from the misuse of advances in AI research are growing (Brundage et al., 2018; Ovadya and Whittlestone, 2019).”