What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment

By Hongyuan Mei, Mohit Bansal, Matthew R. Walter. 2015

293 citations recorded by the source corpus.

We propose an end-to-end, domain-independent neural encoder-aligner-decoder model for selective generation, i.e., the joint task of content selection and surface realization. Our model first encodes a full set of over-determined database event records via an LSTM-based recurrent neural network, then utilizes a novel coarse-to-fine aligner to identify the small subset of salient records to talk about, and finally employs a decoder to generate free-form descriptions of the aligned, selected records. Our model achieves the best selection and generation results reported to-date (with 59% relative improvement in generation) on the benchmark WeatherGov dataset, despite using no specialized features or linguistic resources. Using an improved k-nearest neighbor beam filter helps further. We also perform a series of ablations and visualizations to elucidate the contributions of our key model components. Lastly, we evaluate the generalizability of our model on the RoboCup dataset, and get results that are competitive with or better than the state-of-the-art, despite being severely data-starved.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Generative Models

Citations in the corpus

Semantic Noise Matters for Neural Natural Language Generation (2019, background): “Neural Natural Language Generation (NNLG) is promising for generating text from Meaning Rep-resentations (MRs) in an ‘end-to-end’ fashion, i.e. without needing alignments (Wen et al., 2015, 2016; Duˇsek and Jurˇc´ıˇcek, 2016; Mei et al., 2016).”
The E2E Dataset: New Challenges For End-to-End Generation (2017, background): “…development of NLG components in new domains: They jointly learn sentence planning and surface realisation from non-aligned data (Duˇsek and Jurˇc´ıˇcek, 2015; Wen et al., 2015; Mei et al., 2016; Wen et al., 2016; Sharma et al., 2016; Duˇsek and Jurˇc´ıˇcek, 2016a; Lampouras and Vlachos, 2016).”
DART: Open-Domain Structured Data Record to Text Generation (2020, background): “…Wang, 2019; Taniguchi et al., 2019), weather forecasts (Goldberg et al., 1994; Reiter et al., 2005; Liang et al., 2009; Konstas and Lapata, 2012; Mei et al., 2016), biographical texts (Lebret et al., 2016; Sha et al., 2018; Liu et al., 2018; Vou-giouklis et al., 2018; Perez-Beltrachini and…”
Challenges in Data-to-Document Generation (2017, background): “Mei et al. (2016) use a neural encoder-decoder approach on standard record-based generation datasets, obtaining impressive results, and motivating the need for more challenging NLG problems.”
Neural Text Generation from Structured Data with Application to the Biography Domain (2016, methodology): “Our model is most similar to Mei et al. (2016) who use an encoder-decoder style neural network model to tackle the W EATHERGOV and R OBOCUP tasks.”
Controlling Linguistic Style Aspects in Neural Language Generation (2017, methodology): “Mei et al. (2016) condition the content on a set of key,value pairs using an encoder-decoder architecture with a coarse-to-ﬁne attention mechanism.”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, neutral): “(Dušek and Jurčı́ček, 2015; Wen et al., 2015b; Mei et al., 2016; Wen et al., 2016; Sharma et al., 2016a; Dušek and Jurčı́ček, 2016a; Lampouras and Vlachos, 2016).”
Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings (2016, methodology): “We experiment with using much less training data than recent systems based on recurrent neural networks (RNN) (Wen et al., 2015b; Mei et al., 2015), and we ﬁnd that our generator learns successfully to produce both strings and deep syntax trees on the BAGEL restaurant information dataset (Mairesse…”
Summarizing Source Code using a Neural Attention Model (2016, methodology): “Mei et al. (2016), Konstas and Lapata (2013), and Angeli et al. (2010) create learning algorithms for text generation from database records, again assuming data that pairs sentences with formal meaning representations.”
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning (2017, background): “, 2009, 2010; Vogel and Jurafsky, 2010), and neural network models (Mei et al., 2016).”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “5) have been based on partially aligned data (e.g. Wen et al., 2015; Lebret et al., 2016; Mei et al., 2016).”
Findings of the E2E NLG Challenge (2018, background): “(Dušek and Jurčı́ček, 2015; Wen et al., 2015b; Mei et al., 2016; Wen et al., 2016; Sharma et al., 2016; Dušek and Jurčı́ček, 2016a; Lampouras and Vlachos, 2016).”
Data-to-Text Generation with Content Selection and Planning (2018, methodology): “Recent neural generation systems (Lebret et al. 2016; Mei et al. 2016; Wiseman et al. 2017) do not explicitly model any of these stages, rather they are trained in an end-to-end fashion using the very successful encoder-decoder architecture (Bahdanau et al. 2015) as their backbone.”
End-to-End Content and Plan Selection for Data-to-Text Generation (2018, methodology): “End-to-end methods have already been applied to increasingly complex data to simultaneously learn sentence planning and surface realization but were often restricted by the limited data availability (Wen et al., 2015; Mei et al., 2015; Duˇsek and Jurˇc´ıˇcek, 2016; Lampouras and Vlachos, 2016).”
ESPRIT: Explaining Solutions to Physical Reasoning Tasks (2020, background): “…records (Tanaka-Ishii et al., 1998; Chen and Mooney, 2008; Taniguchi et al., 2019), weather forecasts (Liang et al., 2009; Konstas and Lapata, 2012; Mei et al., 2016), biographical texts from Wikipedia infoboxes (Lebret et al., 2016; Sha et al., 2018; Liu et al., 2018; Perez-Beltrachini and…”
Brain Intelligence: Go beyond Artificial Intelligence (2017, background): “parameters. Long short-term memory architectures (LSTM) [16], which are a further development upon RNN, contain memory cells and multiplicative gates that control the information’s access. Mei et al. [17] proposed a LSTM-based architecture, which uses the encoder-decoder framework, for content selec”