CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities

By Mina Lee, Percy Liang, Qian Yang. 2022

552 citations recorded by the source corpus.

Large language models (LMs) offer unprecedented language generation capabilities and exciting opportunities for interaction design. However, their highly context-dependent capabilities are difficult to grasp and are often subjectively interpreted. In this paper, we argue that by curating and analyzing large interaction datasets, the HCI community can foster more incisive examinations of LMs’ generative capabilities. Exemplifying this approach, we present CoAuthor, a dataset designed for revealing GPT-3’s capabilities in assisting creative and argumentative writing. CoAuthor captures rich interactions between 63 writers and four instances of GPT-3 across 1445 writing sessions. We demonstrate that CoAuthor can address questions about GPT-3’s language, ideation, and collaboration capabilities, and reveal its contribution as a writing “collaborator” under various definitions of good collaboration. Finally, we discuss how this work may facilitate a more principled discussion around LMs’ promises and pitfalls in relation to interaction design. The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

PEER: A Collaborative Language Model (2022, methodology): “Lee et al. (2022) investigate using GPT3 (Brown et al., 2020) as a writing assistant for creative and argumentative writing.”
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022, background): “, 2021) and show potential for creative writing tasks (Lee et al., 2022).”
Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits (2023, background): “On the other hand, rather than steering the language generation with artiﬁcial prompts, humans can easily ﬁx immoral language by making hierarchical and recursive edits (Du et al., 2022; Lee et al., 2022), where human value judgments serve as the guide for each edit.”
Executable Code Actions Elicit Better LLM Agents (2024, methodology): “…trajectories CodeActInstruct for instruction tuning, and train CodeActAgent that is specially designed for seamless integration with Python and can execute sophisticated tasks (e.g., model training) leveraging existing Python packages and autonomously rectifying errors through self-debugging.”
Help me write a Poem - Instruction Tuning as a Vehicle for Collaborative Poetry Writing (2022, background): “, 2022; Branwen, 2020) such as building co-creative assistants for writing stories, poems, and argumentative essays (Lee et al., 2022; Swanson et al., 2021; Uthus et al., 2019; Donahue et al., 2020; Padmakumar and He, 2022; Du et al., 2022).”
Exploring AI Ethics of ChatGPT: A Diagnostic Analysis (2023, methodology): “LLMs have been utilized in various real-world scenarios, including search engines [1, 2], language translation [3, 4], and copywriting [5].”
Instruction Tuning for Large Language Models: A Survey (2023, unknown): “As depicted in Figure 1, it involves further training LLMs using (instruction, output) pairs, where instruction denotes the human instruction for the model, and output denotes the desired output that follows the instruction.”
Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2022, background): “Some studies consulted with expert linguists [31], university students in the sciences [42] or humanities [123], and amateur writers [2, 21, 62, 100, 125].”
Read, Revise, Repeat: A System Demonstration for Human-in-the-loop Iterative Text Revision (2022, methodology): “2); on the other hand, by showing edit intentions for every suggested edit, which users can further decide to accept or reject, R3 provides them with more fine-grained control over the text revision process compared to other one-shot based text revision systems (Lee et al., 2022), and are limited in”