Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models
By Ryan Louie, Andy Coenen, Cheng-Zhi Anna Huang, Michael Terry, Carrie J. Cai. 2020
326 citations recorded by the source corpus.
While generative deep neural networks (DNNs) have demonstrated their capacity for creating novel musical compositions, less attention has been paid to the challenges and potential of co-creating with these musical AIs, especially for novices. In a needfinding study with a widely used, interactive musical AI, we found that the AI can overwhelm users with the amount of musical content it generates, and frustrate them with its non-deterministic output. To better match co-creation needs, we developed AI-steering tools, consisting of Voice Lanes that restrict content generation to particular voices; Example-Based Sliders to control the similarity of generated content to an existing example; Semantic Sliders to nudge music generation in high-level directions (happy/sad, conventional/surprising); and Multiple Alternatives of generated content to audition and choose from. In a summative study (N=21), we discovered the tools not only increased users' trust, control, comprehension, and sense of collaboration with the AI, but also contributed to a greater sense of self-efficacy and ownership of the composition relative to the AI.
Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, LLM Agents & Tool Use, Generative Models, Human-Computer Interaction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Cognitive Neuroscience
- Monica Dinculescu
- Michael J. Muller
- Mary Lou Maher
- Carrie J. Cai
- Jeba Rezwana
- Lydia B. Chilton
- Justin D. Weisz
- Vivian Liu
In Computer Vision and Pattern Recognition
- Monica Dinculescu
- Carrie J. Cai
- Vivian Lai
- Andy Coenen
- Ed Newton-Rex
- Michael Terry
- Michael J. Muller
- Lydia B. Chilton
In LLM Agents & Tool Use
In Generative Models
- Monica Dinculescu
- Andy Coenen
- Carrie J. Cai
- Ann Yuan
- Chenhao Tan
- Cheng-Zhi Anna Huang
- Daphne Ippolito
In Human-Computer Interaction
- Michael Terry
- Tongshuang Sherry Wu
- Vivian Lai
- Michael J. Muller
- Carrie J. Cai
- Lydia B. Chilton
- Justin D. Weisz
- Han Liu
Citations in the corpus
- AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, background): “Contrary to recent work in human-AI interaction, which primarily examined how to increase AI controllability through exposing knobs within a model [44, 49], our work opens up the possibility of steering AI using the model itself as units to control.”
- Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making (2021, background): “AI assistance has been shown to improve people’s efficiency in tasks such as transcription by enhancing their computational capacity [16, 35], support creativity in producing music [15, 41, 45], and even allow the visually impaired to “see” images [22, 68].”
- Design Principles for Generative AI Applications (2024, background): “…classifcation 9 Some user interfaces to generative AI systems allow users to specify their intent via sketches and gestures [32], UI controls [103, 105], video-scanned body movements [174], and even various forms of improv performance [15, 68]. use cases in which false positives and false…”
- Designing Creative AI Partners with COFI: A Framework for Modeling Interaction in Human-AI Co-Creative Systems (2022, background): “AI ability alone does not ensure a positive collaborative experience of users with the AI [100] and interaction is more critical than algorithms where interaction with the users is essential [152].”
- Design Guidelines for Prompt Engineering Text-to-Image Generative Models (2021, background): “[30] introduced CoCoCo, an AI music creation system which lets users move sliders to tune their generated music to be "happier" or "sadder".”
- Wordcraft: Story Writing With Large Language Models (2022, background): “There have been many studies of humanAI co-creation across domains, including drawing [8, 27, 31], music [15, 22, 25], video game content creation [24], and design ideation [16].”
- Human-AI Co-Creation in Songwriting (2020, methodology): “Even when composing short, two-bar counterpoints, it can be challenging for novice musicians to partner with a deep generative model: users desire greater agency, control, and sense of author-ship vis-a-vis the AI during co-creation [45].”