Hierarchical Text Generation and Planning for Strategic Dialogue

By Denis Yarats, Michael Lewis. 2017

61 citations recorded by the source corpus.

End-to-end models for goal-orientated dialogue are challenging to train, because linguistic and strategic aspects are entangled in latent state vectors. We introduce an approach to learning representations of messages in dialogues by maximizing the likelihood of subsequent sentences and actions, which decouples the semantics of the dialogue utterance from its linguistic realization. We then use these latent sentence representations for hierarchical language generation, planning and reinforcement learning. Experiments show that our approach increases the end-task reward achieved by the model, improves the effectiveness of long-term planning using rollouts, and allows self-play reinforcement learning to improve decision making without diverging from human language. Our hierarchical latent-variable model outperforms previous work both linguistically and strategically.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Hierarchical Neural Story Generation (2018, background): “Yarats and Lewis (2017) generate a discrete latent variable based on the context, then generates text conditioned upon it.”
Reason first, then respond: Modular Generation for Knowledge-infused Dialogue (2021, methodology): “In reinforcement learning settings, generating natural language has been used as an intermediate planning step (Sharma et al., 2021; Hu et al., 2019), and in particular in goal-oriented dialogue (Yarats and Lewis, 2018) and open-domain QA (Adolphs et al., 2021) as well.”
Hierarchical Decision Making by Generating and Following Natural Language Instructions (2019, unknown): “Previous work has used language to specify exponentially many policies [14, 1, 27], allowing zero-shot generalization across tasks.”
Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation (2019, methodology): “, 1977) is commonly used to evaluate the quality of a language model, which has also been employed to evaluate generators (Yarats and Lewis, 2018), (Ficler and Goldberg, 2017), (Gerz et al.”
Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog (2019, background): “Among many models, Reinforcement Learning (RL) is commonly used to learn dialog policy (Fatemi et al., 2016; Peng et al., 2017; Chen et al., 2017; Yarats and Lewis, 2018; Lei et al., 2018; He et al., 2018; Su et al., 2018), where users are modeled as a part of the environment and the policy is…”
Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models (2019, background): “or learned, to decouple the word generation process from dialog policy in E2E systems for better dialog policy (He et al., 2018; Yarats and Lewis, 2017).”
Recommendation as a Communication Game: Self-Supervised Bot-Play for Goal-oriented Dialogue (2019, unknown): “, 2017a), and semi-cooperative negotiation tasks (Lewis et al., 2017; Yarats and Lewis, 2018; He et al., 2018).”
Improving Factual Consistency Between a Response and Persona Facts (2021, background): “As also suggested in prior work (Yarats and Lewis, 2018; Zhao et al., 2019; Bao et al., 2019), applying RL for speciﬁc metrics might bring in adverse impacts on linguistic quality.”
How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds (2020, background): “In particular, the negotiation tasks of (Yarats and Lewis, 2017; Lewis et al., 2017), where two agents are trying to convince each other to perform certain actions, are related to the tasks in LIGHT-Quests.”
Situated Dialogue Learning through Procedural Environment Generation (2021, background): “(2021), the negotiation tasks of (Yarats and Lewis, 2017; Lewis et al., 2017), where two agents are trying to convince each other to perform certain actions, are related to the tasks in LIGHT-Quests.”
Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation (2018, background): “There has been some recent efforts [42] devoted in applying hierarchical reinforcement learning (HRL) [7] where sequence generation is broken down into several sub-tasks each of which targets at a chunk of words.”
MAKING KNIGHTS SMILE IN A FANTASY GAME WORLD (2019, methodology): “A solution to both problems is to train most of the parameters of the model with human-human language data, and then to either disentangle or only optimize some of the parameters with model self-chat (Yarats & Lewis, 2017).”
I love your chain mail! Making knights smile in a fantasy game world: Open-domain goal-oriented dialogue agents (2019, background): “A related approach to ours is the negotiation task of Lewis et al. (2017); Yarats & Lewis (2017), which requires two agents to swap 3 item types (hats, balls, books) where the value of the items is different for the two agents, and derives their personal reward.”