Crowd-sourcing NLG Data: Pictures Elicit Better Data.

By Jekaterina Novikova, Oliver Lemon, Verena Rieser. 2016

74 citations recorded by the source corpus.

Recent advances in corpus-based Natural Language Generation (NLG) hold the promise of being easily portable across domains, but require costly training data, consisting of meaning representations (MRs) paired with Natural Language (NL) utterances. In this work, we propose a novel framework for crowdsourcing high quality NLG training data, using automatic quality control measures and evaluating different MRs with which to elicit data. We show that pictorial MRs result in better NL data being collected than logic-based MRs: utterances elicited by pictorial MRs are judged as significantly more natural, more informative, and better phrased, with a significant increase in average quality ratings (around 0.5 points on a 6-point scale), compared to using the logical MRs. As the MR becomes more complex, the benefits of pictorial stimuli increase. The collected data will be released as part of this submission.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Semantic Noise Matters for Neural Natural Language Generation (2019, background): “However, NNLG requires large volumes of in-domain data, which is typically crowdsourced (e.g. Mairesse et al., 2010; Novikova et al., 2016; Wen et al., 2015, 2016; Howcroft et al., 2017), introducing noise.”
The E2E Dataset: New Challenges For End-to-End Generation (2017, methodology): “The data was collected using the Crowd-Flower platform and quality-controlled following Novikova et al. (2016).”
Why We Need New Evaluation Metrics for NLG (2017, background): “(Novikova et al., 2016), and as such, enable rapid development of NLG components in new domains.”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, background): “Target Quality Target Source Content Selection Wikibio (Lebret et al., 2016) 583K Biographies Noisy Wikipedia Not speciﬁed Rotowire (Wiseman et al., 2017) 4.9K Basketball Noisy Rotowire Not speciﬁed WebNLG (Gardent et al., 2017b) 25.3K 15 DBPedia categories Clean Annotator Generated Fully speciﬁed E”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, methodology): “Recent research often adds a general, overall quality criterion (Wen et al. 2015b,a; Manishina et al. 2016; Novikova, Lemon, and Rieser 2016; Novikova et al. 2017), or even uses only that (Sharma et al. 2016a).”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “Recent proposals for new challenges extend these to narrative generation (Concepción et al., 2016), generation from structured web data (Colin et al., 2016) and from unaligned pairs of meaning representations and text (Novikova & Rieser, 2016a).”
RankME: Reliable Human Ratings for Natural Language Generation (2018, background): “This is one of the reasons why some more recent research adds a general, overall quality criterion (Wen et al., 2015a,b; Manishina et al., 2016; Novikova et al., 2016, 2017a), or even uses only that (Sharma et al., 2016).”
Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation (2017, methodology): “The task-oriented dialogue community should move towards using larger and more complex datasets, which have been recently announced, such as the Frames dataset (El Asri et al., 2017) or the E2E NLG Challenge dataset (Novikova et al., 2016).”
Findings of the E2E NLG Challenge (2018, background): “In a pre-study (Novikova et al., 2016), we showed that pictorial MRs provide similar collection speed and utterance length, but are less likely to prime the crowd workers in their lexical choices.”
Evaluation of Text Generation: A Survey (2020, neutral): “, 2017), e2e (Novikova et al., 2016), wikibio (Lebret et al.”
A Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generation (2018, background): “…were Note that the number of MRs in the E2E dataset was cut off at 10K for the sake of visibility of the small differences between other column pairs. collected using pictures as the source of information, which was shown to inspire more informative and natural utterances (Novikova et al., 2016).”
TNT-NLG , System 2 : Data Repetition and Meaning Representation Manipulation to Improve Neural Generation (2018, background): “The data from this challenge is within the restaurant domain, where 50k instances have been crowdsourced and have an open vocabulary, complex syntactic structures and diverse discourse phenomena (Novikova et al., 2016).”
Controlling Personality-Based Stylistic Variation with Neural Natural Language Generators (2018, unknown): “This requires training data that allows the NLG to learn how to map semantic representations for system dialogue acts to one or more possible outputs (see Figure 1, (Novikova et al., 2016)).”
The aNALoGuE Challenge: Non Aligned Language GEneration (2016, methodology): “Furthermore, our crowd-sourced utterances were elicited using pictures, which makes them more varied in sentence structure and vocabulary than previously used corpora (Novikova and Rieser, 2016).”