Deal or No Deal? End-to-End Learning of Negotiation Dialogues

By Michael Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, Dhruv Batra. 2017

499 citations recorded by the source corpus.

Much of human dialogue occurs in semi-cooperative settings, where agents with different goals attempt to agree on common decisions. Negotiations require complex communication and reasoning skills, but success is easy to measure, making this an interesting task for AI. We gather a large dataset of human-human negotiations on a multi-issue bargaining task, where agents who cannot observe each other’s reward functions must reach an agreement (or a deal) via natural language dialogue. For the first time, we show it is possible to train end-to-end models for negotiation, which must learn both linguistic and reasoning skills with no annotated dialogue states. We also introduce dialogue rollouts, in which the model plans ahead by simulating possible complete continuations of the conversation, and find that this technique dramatically improves performance. Our code and dataset are publicly available.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Experience Grounds Language (2020, background): “In order to learn the effects language has on the world, an agent must participate in linguistic activity, such as negotiation (Yang et al., 2019a; He et al., 2018; Lewis et al., 2017), collaboration (Chai et al.”
QuAC: Question Answering in Context (2018, background): “More explicit goal driven scenarios, such as bargaining (Lewis et al., 2017)”
Ethical and social risks of harm from Language Models (2021, background): “In a NLP setup where two RL agents negotiate using natural language, (Lewis et al., 2017) found ‘agents have learnt to deceive without any explicit human design, simply by trying to achieve their goals’.”
Towards Exploiting Background Knowledge for Building Conversation Systems (2018, neutral): “We would also like to mention some existing works such as (He et al., 2017; Lewis et al., 2017; Krause et al., 2017) which have used crowdsourcing for creating conversation datasets.”
Alignment of Language Agents (2021, background): “Lewis et al. (2017) found that their negotiation agent learnt to deceive from self-play, without any explicit human design.”
Towards Coherent and Cohesive Long-form Text Generation (2018, background): “We also continue the supervised learning of G to constrain the policy search within a space that represents the sentences that are likely to be grammatically plausible, similar to Paulus et al. (2017); Wu et al. (2016); Lewis et al. (2017).”
OpenSpiel: A Framework for Reinforcement Learning in Games (2019, neutral): “Game Reference(s) Backgammon Wikipedia Breakthrough Wikipedia Bridge bidding Wikipedia Catch [51] and [59, Appendix A] Coin Game [64] Connect Four Wikipedia Cooperative Box-Pushing [70] Chess Wikipedia First-price Sealed-bid Auction Wikipedia Go Wikipedia Goofspiel Wikipedia Hanabi (via HLE) Wikiped”
Neural Approaches to Conversational AI (2018, neutral): “l. (2018) described such a dialogue self-play procedure, which can generate conversations between a simulated user and the system agent. Promising results have been observed in negotiation dialogues (Lewis et al., 2017) and task-oriented dialogues (Liu and Lane, 2017; Shah et al., 2018; Wei et al., ”
The Hanabi Challenge: A New Frontier for AI Research (2019, background): “Algorithms have been developed to learn to solve riddles and referential games [6,73], gridworld games requiring coordination [74], object identification via question-and-answer dialog [75], and negotiation [7,8].”
Ask the Right Questions: Active Question Reformulation with Reinforcement Learning (2017, neutral): “et, AQA-QR signiﬁcantly outperforms all others in the QA task. Training the agent starting from the Base-NMT+Quora model yielded comparable results as starting from Base-NMT. 6.3 DISCUSSION Recently, Lewis et al. (2017) trained chatbots that negotiate via language utterances in order to complete a t”
Emergent Communication through Negotiation (2018, methodology): “A related paper from Lewis et al. (2017) takes a top-down approach to learning to negotiate by leveraging dialogue data.”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, neutral): “ deﬁned three situations where the bot has problems in answering a question. Compared the experimental results of not using a askingquestion way, this method made great improvement in some scenarios. [37] explored the task on negotiation dialogues. As conventional sequence-to-sequence models simulat”
Taxonomy of Risks posed by Language Models (2022, background): “It has already been observed that RL agents could, in principle, learn such techniques: in one NLP study where two RL agents negotiate using natural language, ‘agents have learnt to deceive without any explicit human design, simply by trying to achieve their goals’ [114].”
Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints (2018, neutral): “Lewis et. al.”
I love your chain mail! Making knights smile in a fantasy game world: Open-domain goal-oriented dialogue agents (2019, unknown): “A related approach to ours is the negotiation task of Lewis et al. (2017); Yarats & Lewis (2017), which requires two agents to swap 3 item types (hats, balls, books) where the value of the items is different for the two agents, and derives their personal reward.”
Prosocial learning agents solve generalized Stag Hunts better than selfish ones (2017, background): “…can interact with others (Silver et al. 2016; Lazaridou, Peysakhovich, and Baroni 2017; Das et al. 2017; Lowe et al. 2017; Foer-ster et al. 2017; Evtimova et al. 2017; Foerster et al. 2016; Havrylov and Titov 2017; Lewis et al. 2017; Heinrich and Silver 2016; Peng et al. 2017; Leibo et al. 2017).”
Reward Design with Language Models (2023, methodology): “We closely follow the implementation outlined in (Kwon et al., 2021; Lewis et al., 2017), please refer to those papers for more training details.”
Recommendation as a Communication Game: Self-Supervised Bot-Play for Goal-oriented Dialogue (2019, background): “Other dialogue settings with shared objectives have been explored, for example a collaborative graph prediction task (He et al., 2017a), and semi-cooperative negotiation tasks (Lewis et al., 2017; Yarats and Lewis, 2018; He et al., 2018).”
MOSS: End-to-End Dialog System Framework with Modular Supervision (2019, background): “Towards tackling tasks with more dialog acts, Lewis et al. (2017); He et al. (2018); Wang et al. (2019) collected negotiation and persuasion dialogs by asking the two Turkers negotiate or persuade each other to reach an agreement.”