Semantic Noise Matters for Neural Natural Language Generation

By Ondrej Dusek, David M. Howcroft, Verena Rieser. 2019

124 citations recorded by the source corpus.

Neural natural language generation (NNLG) systems are known for their pathological outputs, i.e. generating text which is unrelated to the input specification. In this paper, we show the impact of semantic noise on state-of-the-art NNLG models which implement different semantic control mechanisms. We find that cleaned data can improve semantic correctness by up to 97%, while maintaining fluency. We also find that the most common error is omitting information, rather than hallucination.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “[cz] • WebNLG 2020 (Gardent et al., 2017; Castro Ferreira et al., 2020) – Verbalize subject-predicate-object triples in one or more sentences in a grammatical and natural way.”
Finetuned Language Models Are Zero-Shot Learners (2021, neutral): “E2ENLG (Dušek et al., 2019) 4.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “E2E (cleaned) (Novikova et al., 2017; Dušek et al., 2019) 11.”
Survey of Hallucination in Natural Language Generation (2022, background): “Considering this issue, other works [37, 137] correct paired training samples, specifically the input data, according to the references.”
DART: Open-Domain Structured Data Record to Text Generation (2020, methodology): “…3.1), (2) automatic conversion of questions in WikiSQL to declarative sentences (Section 3.2), and (3) incorporation of existing datasets including WebNLG 2017 (Gardent et al., 2017a,b; Shi-morina and Gardent, 2018) and Cleaned E2E (Novikova et al., 2017b; Dušek et al., 2018, 2019) (Section 3.3).”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “, 2020), Clean E2E NLG (Dušek et al., 2019) and WikiLingua (Scialom et al.”
Control Prefixes for Parameter-Efficient Text Generation (2021, methodology): “…outperforms previous approaches, many of them based on full fine-tuning, when evaluated on the WebNLG (Gardent et al., 2017), WebNLG+ 2020 (Castro Ferreira et al., 2020), DART (Radev et al., 2020) and E2E Clean (Dušek et al., 2019) data-totext datasets using the official evaluation scripts.”
Unifying Language Learning Paradigms (2022, methodology): “We use CNN/Dailymail (Hermann et al., 2015), XSUM (Narayan et al., 2018), MultiNews (Fabbri et al., 2019), SAMSum (Gliwa et al., 2019), WebNLG (Castro Ferreira et al., 2020) (English), E2E (Duˇsek et al., 2019) and CommonGen (Lin et al., 2020) to evaluate our models.”
TaTa: A Multilingual Table-to-Text Dataset for African Languages (2022, methodology): “Noise in simpler datasets can often be detected and ﬁltered out through regular expressions (Reed et al., 2018), as done by Dušek et al. (2019) for the E2E dataset (Novikova et al., 2017).”
The Third Multilingual Surface Realisation Shared Task (SR’20): Overview and Evaluation Results (2020, background): “There is much discussion in the field of how to control the vexed tendencies of neural generators to ‘hallucinate’ content (Dušek et al., 2019), and how to instil some order”
A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems (2020, methodology): “Semantic control techniques for bringing the data into closer alignment with the text (Dušek et al., 2019) have been shown to be useful with less complex datasets such as the E2E Challenge (Dušek et al., 2018).”
Text Generation with Efficient (Soft) Q-Learning (2021, methodology): “A.1 Setup Details Datasets We use four datasets: E2E [40, 11], CommonGen [32], SNLI [3], and MultiNLI [61].”
Automatic Construction of Evaluation Suites for Natural Language Generation Datasets (2021, background): “Languages Communicative Goal Reference Data-to-text CommonGen CG en Expand concept words into natural language [32] Czech Restaurant CR cs Verbalize an agents response from dialog act [33] E2E E2 en Describe a restaurant from key-value attributes [34] ToTTo TO en Describe highlighted cells in a tabl”
Controlling hallucinations at word level in data-to-text generation (2021, unknown): “Indeed, rule-based [9] or neural-based methods [41] have been proposed, speciﬁcally with table regularization techniques, where attributes are added or removed to re-align table and target descrip-Controlling tion.”
Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data (2020, unknown): “It has often been observed that hallucinations can be partially explained by a strong LM component in the decoder which tends to select the next token as a likely continuation of the sequence generated so far (Rohrbach et al., 2018; Duˇsek et al., 2019, inter alia).”