Controlling Output Length in Neural Encoder-Decoders

By Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, Manabu Okumura. 2016

254 citations recorded by the source corpus.

Neural encoder-decoder models have shown great success in many sequence generation tasks. However, previous work has not investigated situations in which we would like to control the length of encoder-decoder outputs. This capability is crucial for applications such as text summarization, in which we have to generate concise summaries with a desired length. In this paper, we propose methods for controlling the output sequence length for neural encoder-decoder models: two decoding-based methods and two learning-based methods. Results show that our learning-based methods have the capability to control length without degrading summary quality in a summarization task.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “One solution is the use of controllable neural text generation methods, in particular conditional training (Fan, Grangier, and Auli, 2018; Kikuchi et al., 2016; Peng et al., 2018) and weighted decoding (Ghazvininejad et al., 2017; Baheti et al., 2018).”
Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, methodology): “To create models that can generate a gender-balanced number of gendered words, we propose Conditional Training (CT) for controlling generative model output (Kikuchi et al., 2016; Fan et al., 2017; Oraby et al., 2018; See et al., 2019).”
Plug and Play Language Models: A Simple Approach to Controlled Text Generation (2019, methodology): “, 2017), or training conditional generative models (Kikuchi et al., 2016; Ficler & Goldberg, 2017).”
Neural Text Generation with Unlikelihood Training (2019, background): “Some other lines of work of note are the use of generative control (Kikuchi et al., 2016; Fan et al., 2017; Peng et al., 2018; See et al., 2019) and the retrieve-and-reﬁne setting (Weston et al., 2018; Guu et al., 2018).”
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, background): “Speciically, the guidance signals could be style tokens [27, 123], length speciications [56], domain tags [14], or any variety of other pieces of information used to control of the generated text.”
Controllable Abstractive Summarization (2017, methodology): “(2016), our results are stronger while our method is very simple – (Kikuchi et al., 2016) explore embedding the remaining length at each timestep during decoding and creating a separate memory cell to control length.”
What makes a good conversation? How controllable attributes affect human judgments (2019, methodology): “Given these constraints, we study two con-trol methods: conditional training (Fan et al., 2018; Kikuchi et al., 2016; Peng et al., 2018) and weighted decoding (Ghazvininejad et al., 2017).”
Controlling Linguistic Style Aspects in Neural Language Generation (2017, background): “, 2016); the politeness of the generated text or the text length in a machinetranslation setting (Sennrich et al., 2016a; Kikuchi et al., 2016); or the tense in generated movie reviews (Hu et al.”
Explaining Question Answering Models through Text Generation (2020, background): “However, generation is not controlled by additional inputs (Kikuchi et al., 2016; Ficler and Goldberg, 2017; Keskar et al., 2019; Dathathri et al., 2019).”
Cutting-off Redundant Repeating Generations for Neural Abstractive Summarization (2016, methodology): “In fact, this model has already been used as a strong baseline for ABS tasks Chopra et al. (2016); Kikuchi et al. (2016) as well as in the NMT literature.”
Steering Output Style and Topic in Neural Response Generation (2017, background): “, 2015), summarization (Rush et al., 2015; Gu et al., 2016; Kikuchi et al., 2016), and conversation generation (Vinyals and Le, 2015; Sordoni et al.”
Multi-style Generative Reading Comprehension (2019, methodology): “The models proposed by McCann et al. (2018), Bauer et al.”
Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data (2020, unknown): “This can be achieved by reading the length off the training data and using it as an additional input during training so that during inference one obtains a ”length knob” (Kikuchi et al., 2016, Fig.”
Control Prefixes for Parameter-Efficient Text Generation (2021, background): “Controlled generation aims to incorporate various types of guidance beyond the input text into the generation model (Kikuchi et al., 2016).”
CoCon: A Self-Supervised Approach for Controlled Text Generation (2020, background): “Some of the earliest efforts involve conditional generative models [16, 8] where the networks are trained on text data labeled with the target attributes.”
Recent advances in document summarization (2017, background): “As it is difficult to pose hard constraints on decoder generation, one recent work [89] studies various solutions, including direct truncation on generated sequence, discarding out-of-range generations in the decoding beam and directly embedding length information in the LSTM units.”
Plug-and-Play Conversational Models (2020, background): “In terms of controlled dialogue generation, See et al. (2019) studied of conditional generative models (Kikuchi et al., 2016) and weighted decoding (Ghazvininejad et al., 2017) in controlling models trained on persona-chat.”