Task-Oriented Dialogue as Dataflow Synthesis

By Jacob Andreas, J. Bufe, David Burkett, Charles C. Chen, Joshua Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner and 34 more. 2020

176 citations recorded by the source corpus.

Abstract We describe an approach to task-oriented dialogue in which dialogue state is represented as a dataflow graph. A dialogue agent maps each user utterance to a program that extends this graph. Programs include metacomputation operators for reference and revision that reuse dataflow fragments from previous turns. Our graph-based state enables the expression and manipulation of complex user intents, and explicit metacomputation makes these intents easier for learned models to predict. We introduce a new dataset, SMCalFlow, featuring complex dialogues about events, weather, places, and people. Experiments show that dataflow graphs and metacomputation substantially improve representability and predictability in these natural dialogues. Additional experiments on the MultiWOZ dataset show that our dataflow representation enables an otherwise off-the-shelf sequence-to-sequence model to match the best existing task-specific state tracking model. The SMCalFlow dataset, code for replicating experiments, and a public leaderboard are available at https://www.microsoft.com/en-us/research/project/dataflow-based-dialogue-semantic-machines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis (2022, background): “Several works investigate converting conversational intents into programmable representations, such as SQL (Yu et al., 2019a;b) or dataflow graph (Andreas et al., 2020).”
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, background): “These include: 1) generation of good search queries [22, 59] and reformulation [37, 51], 2) strategic exploration for navigating through the website [55, 56, 29], 3) robust language understanding for textual state and action spaces [3, 7, 17, 44], and 4) long-term”
Learning To Retrieve Prompts for In-Context Learning (2021, background): “In this setting, we evaluate on the full development sets of B REAK , MT OP , and SMC AL F LOW .”
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction (2021, background): “They contrast the task with conversational semantic parsing (Suhr et al., 2018; Yu et al., 2019b,a; Andreas et al., 2020) and show that the two tasks are distinct and are addressing different aspects of utilizing context.”
Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning (2021, background): “It is therefore more closely related to systems which extract programs from dialogue, such as Andreas et al. (2020).”
τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025, unknown): “Following a long line of research into language agents [25, 27, 10, 13, 21], LLM tool use [24, 20, 9], and task-oriented dialog [5, 4, 1, 22, 6, 7, 8], τ -bench [26] is a recently introduced benchmark to measure the reliability of language agents in multi-turn task-oriented conversations such as…”
Synchromesh: Reliable code generation from pre-trained language models (2022, methodology): “For SMCalFlow, we use the dataset that introduced the language (Andreas et al., 2020).”
Value-Agnostic Conversational Semantic Parsing (2021, methodology): “Our model substantially improves upon the best published results on the SMC AL F LOW (Semantic Machines et al., 2020) and T REE DST (Cheng et al., 2020) conversational semantic parsing datasets, improving model performance by 7.3% and 10.6%, respectively, in terms of absolute accuracy.”
ReasonBERT: Pre-trained to Reason with Distant Supervision (2021, background): “…over multiple pieces of evidence, hybrid QA (Chen et al., 2020) requires simultaneously reasoning over un-structured text and structured tables, and dialogue systems require reasoning over the whole dialogue history to accurately understand the current user utterance (Andreas et al., 2020).”
Neurosymbolic Program Synthesis (unknown): “For example, Andreas et al. [20] design a calendar assistant in which scheduling actions, dialog corrections, and exceptions are represented as compositional programs, synthesized by neurosymbolic models in context.”
L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models (2023, background): “As illustrated in Figure 1, language-to-code is the foundation of many applications in AI, such as task-oriented dialogue systems (Andreas et al., 2020), coding assistant (Agashe et al., 2019; Lai et al., 2023), language interfaces to databases (Pasupat and Liang, 2015; Yu et al., 2018), and…”
Programming Puzzles (2021, background): “return state == [3] * num_disks # All disks must end on tower 3.”
Natural Language to Code Generation in Interactive Data Science Notebooks (2022, methodology): “Next, for the following problems after cell 4, we include reference solutions to previous problems in their prompts, following the multi-turn task-oriented dialogue setting in Andreas et al. (2020).”
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024, background): “The former types of benchmarks are static and only test the conversational agent on pre-collected conversation trajectories[4, 3, 2].”
TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems (2020, background): “Earlier work by Andreas et al. (2020) and Hosseini-Asl et al. (2020) employs a similar modeling approach to predict dialog state in task-based dialogs, which can be seen as a precursor to our API call prediction strategy.”