Human-AI Co-Creation in Songwriting

By Cheng-Zhi Anna Huang, Hendrik Vincent Koops, Ed Newton-Rex, Monica Dinculescu, Carrie J. Cai. 2020

109 citations recorded by the source corpus.

Machine learning is challenging the way we make music. Although research in deep generative models has dramatically improved the capability and fluency of music models, recent work has shown that it can be challenging for humans to partner with this new class of algorithms. In this paper, we present findings on what 13 musician/developer teams, a total of 61 users, needed when co-creating a song with AI, the challenges they faced, and how they leveraged and repurposed existing characteristics of AI to overcome some of these challenges. Many teams adopted modular approaches, such as independently running multiple smaller models that align with the musical building blocks of a song, before re-combining their results. As ML models are not easily steerable, teams also generated massive numbers of samples and curated them post-hoc, or used a range of strategies to direct the generation, or algorithmically ranked the samples. Ultimately, teams not only had to manage the "flare and focus" aspects of the creative process, but also juggle them with a parallel process of exploring and curating multiple ML models and outputs. These findings reflect a need to design machine learning-powered music interfaces that are more decomposable, steerable, interpretable, and adaptive, which in return will enable artists to more effectively explore how AI can extend their personal expression.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Carrie J. Cai
2. Michael Terry
3. Tongshuang Sherry Wu
4. Negar Rostamzadeh
5. Ramya Srinivasan
6. Andy Coenen
7. Lydia B. Chilton
8. Vivian Liu

In Computer Vision and Pattern Recognition

1. Negar Rostamzadeh
2. Ramya Srinivasan
3. Andy Coenen
4. Michael Terry
5. Carrie J. Cai
6. Lydia B. Chilton
7. Vivian Liu
8. Ann Yuan

In Multimodal Learning

1. Tongshuang Sherry Wu
2. Emily L. Denton

In Signal Processing

1. Carrie J. Cai
2. Emily L. Denton

Citations in the corpus

• AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, background): “Prior work has noted core challenges of the interaction, such as a lack of transparency, controllability, and user agency [5, 13, 31].”
• Design Guidelines for Prompt Engineering Text-to-Image Generative Models (2021, background): “In recent years, systems embedding generative models have been successfully applied to domains such as image generation, poetry, and music.[14, 15, 18, 19]”
• Wordcraft: Story Writing With Large Language Models (2022, background): “There have been many studies of humanAI co-creation across domains, including drawing [8, 27, 31], music [15, 22, 25], video game content creation [24], and design ideation [16].”
• Artsheets for Art Datasets (2021, background): “[71] Cheng-Zhi Anna Huang, Hendrik Vincent Koops, Ed Newton-Rex, Monica Dinculescu, and Carrie J Cai.”