AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts
By Tongshuang Sherry Wu, Michael Terry, Carrie J. Cai. 2021
679 citations recorded by the source corpus.
Although large language models (LLMs) have demonstrated impressive potential on simple tasks, their breadth of scope, lack of transparency, and insufficient controllability can make them less effective when assisting humans on more complex tasks. In response, we introduce the concept of Chaining LLM steps together, where the output of one step becomes the input for the next, thus aggregating the gains per step. We first define a set of LLM primitive operations useful for Chain construction, then present an interactive system where users can modify these Chains, along with their intermediate results, in a modular way. In a 20-person user study, we found that Chaining not only improved the quality of task outcomes, but also significantly enhanced system transparency, controllability, and sense of collaboration. Additionally, we saw that users developed new ways of interacting with LLMs through Chains: they leveraged sub-tasks to calibrate model expectations, compared and contrasted alternative strategies by observing parallel downstream effects, and debugged unexpected model outputs by “unit-testing” sub-components of a Chain. In two case studies, we further explore how LLM Chains may be used in future applications.
Subfields: Artificial Intelligence, Computer Science Applications, Information Systems, Prompting & Reasoning, LLM Agents & Tool Use
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Quoc V. Le
- Jianfeng Gao
- Yejin Choi
- Graham Neubig
- Yoshua Bengio
In Computer Science Applications
- Mihir Parmar
- Michael S. Bernstein
- Swaroop Mishra
- Chitta Baral
- Quoc V. Le
- Percy Liang
- Thomas Scialom
- Timo Schick
In Information Systems
In Prompting & Reasoning
In LLM Agents & Tool Use
- Swaroop Mishra
- Mihir Parmar
- Chitta Baral
- Man Luo
- Percy Liang
- Qian Yang
- J.D. Zamfirescu-Pereira
- Bjoern Hartmann
Citations in the corpus
- Emergent Abilities of Large Language Models (2022, neutral): “The TruthfulQA benchmark (Lin et al., 2021) showed that GPT-3 models were more likely to mimic human falsehoods as they got larger, though Rae et al. (2021) later showed on a multiple-choice version that scaling Gopher to 280B enabled emergent performance substantially better than random.”
- Augmented Language Models: a Survey (2023, methodology): “…al., 2019), or solve the sub-problems sequentially, where the solution to the next sub-problem depends on the answer to the previous ones (Yang et al., 2022a; Zhou et al., 2022; Drozdov et al., 2022; Dua et al., 2022; Khot et al., 2022; Wang et al., 2022a; Wu et al., 2022b; Mishra and Nouri, 2022).”
- Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (2022, background): “Wu et al. (2022) propose chaining large language model steps such that the output of one step becomes the input for the next and develop an interactive system for users to construct and modify chains.”
- Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022, neutral): “Wu et al. (2022) chain PLM outputs and inputs.”
- CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022, background): “The most traditional approach is contextual inquiry, inviting writers to write with an LM and interviewing them afterward [10, 14, 26, 63, 64, 66].”
- In-BoXBART: Get Instructions into Biomedical Multi-Task Learning (2022, methodology): “Weller et al. (2020) and Ye and Ren (2021a) for improving task instructions for large LMs. Recently, many approaches have been proposed to improve model performance using instructions (Wu et al., 2021, 2022; Lin et al., 2021; Kuznia et al., 2022).”
- Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts (2023, methodology): “In this work, we use these previously-identifed challenges to better understand why we observe some of the struggles with end-user prompt engineering that we do.”
- Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2022, methodology): “Dramatron leverages the strengths of LLMs and combines custom prompts (see Appendix E) and prompt chaining [121] with structured generation for long range coherence across the entire script.”
- Multi-Agent Collaboration Mechanisms: A Survey of LLMs (2025, neutral): “Similarly, sequential channel aggregation is explored in other works [135, 138], where collaboration channels are connected in a sequence to amplify the benefits of individual channels.”
- How Many Data Samples is an Additional Instruction Worth? (2022, neutral): “In addition, many approaches have been proposed to improve model performance us-ing instructions (Wu et al., 2022; Lin et al., 2021; Wang et al., 2022b; Luo et al., 2022; Kuznia et al., 2022; Patel et al., 2022; Mishra and Nouri, 2022).”
- Decomposition Enhances Reasoning via Self-Evaluation Guided Decoding (2023, neutral): “Various decomposition techniques have been explored, such as question reduction (Zhou et al., 2022; Yang et al., 2022), iterative prompting (Wang et al., 2022a), and chaining the steps (Wu et al., 2022).”
- AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “Indeed, transparency and control have long been studied together in HCI as intertwined design goals for effective user experience (M. K. Lee et al., 2019; Wu et al., 2022).”
- Ask Me Anything: A simple strategy for prompting language models (2022, methodology): “The original format of a task varies widely and prior works manually rewrite input examples to new formats in a task-specific manner [Mishra et al., 2021, Wu et al., 2022], which is challenging to scale.”
- AI literacy and its implications for prompt engineering strategies (2024, background): “Contradictory task instructions within the context have also been discovered, even though the extended context in prompts has been demonstrated to improve text outputs (Wu, Terry, & Cai, 2021).”
- Large language models are few-shot clinical information extractors (2022, methodology): “, first asking if a certain entity type exists in the input, before asking for a list (Li et al., 2019b; Wu et al., 2022) or (ii) using an output structure like the sequence tagging approach.”