Eval all, trust a few, do wrong to none: Comparing sentence generation models

By Ondřej Cífka, Aliaksei Severyn, Enrique Alfonseca, Katja Filippova. 2018

41 citations recorded by the source corpus.

In this paper, we study recent neural generative models for text generation related to variational autoencoders. Previous works have employed various techniques to control the prior distribution of the latent codes in these models, which is important for sampling performance, but little attention has been paid to reconstruction error. In our study, we follow a rigorous evaluation protocol using a large set of previously used and novel automatic and human evaluation metrics, applied to both generated samples and reconstructions. We hope that it will become the new evaluation standard when comparing neural generative models for text.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

The Curious Case of Neural Text Degeneration (2019, background): “A number of metrics (based on BLEU and cross entropy) have been proposed to quantify the diversity and quality of open-ended generations (Caccia et al., 2018; Zhu et al., 2018; Cfka et al., 2018).”
Assessing Generative Models via Precision and Recall (2018, methodology): “Following [6], we embed these sentences using a BiLSTM with 2048 cells in each direction and max pooling, leading to a 4096-dimensional embedding [7].”
On Accurate Evaluation of GANs for Language Generation (2018, background): “Another related work conducts a study on the properties of Variational and Adversarial Autoencoder-based generative models [9].”
On Variational Learning of Controllable Representations for Text without Supervision (2019, background): “Cı́fka et al. (2018); Zhao et al. (2018) observed that text manipulation is generally more challenging compared to images, and the successes of these models cannot be directly transferred to texts.”
The Vendi Score: A Diversity Evaluation Metric for Machine Learning (2022, methodology): “…one ﬁt to the embeddings of the sample to be evaluated for diversity. ﬁd was originally proposed for evaluating image generative adversarial networks (gans) but has since been applied to text (Cífka et al., 2018) and molecules (Preuer et al., 2018) using domain-speciﬁc neural network encoders.”
Adversarially Regularized Autoencoders (2017, methodology): “Cífka et al. (2018) recently evaluated a suite of sentence generation models and found that models are quite sensitive to their training setup, and that different models do well on different metrics.”
Learning Latent Space Energy-Based Prior Model (2020, methodology): “To evaluate the quality of the generated samples, we follow [63, 9] and recruit Forward Perplexity (FPPL) and Reverse Perplexity (RPPL).”
A Discriminator Improves Unconditional Text Generation without Updating the Generator (2020, unknown): “(Cı́fka et al., 2018) think BLEU and self-BLEU only focus on local consistency and designs a language model score and a reverse language model score for evaluating generated text on global semantics.”