Texygen: A Benchmarking Platform for Text Generation Models

By Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, Yong Yu. 2018

897 citations recorded by the source corpus.

We introduce Texygen, a benchmarking platform to support research on open-domain text generation models. Texygen has not only implemented a majority of text generation models, but also covered a set of metrics that evaluate the diversity, the quality and the consistency of the generated texts. The Texygen platform could help standardize the research on text generation and improve the reproductivity and reliability of future research work in text generation.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

The Curious Case of Neural Text Degeneration (2019, background): “A number of metrics (based on BLEU and cross entropy) have been proposed to quantify the diversity and quality of open-ended generations (Caccia et al., 2018; Zhu et al., 2018; Cfka et al., 2018).”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, methodology): “Consistently with [23], we measure these two aspects using, respectively, BLEU [29] and self-BLEU [60] metrics.”
Red Teaming Language Models with Language Models (2022, methodology): “When red teaming for offensive replies (§3 and Appendix §C.1), we measured the diversity of generated test cases using Self-BLEU, which may be limited as an automatic metric.”
Aligning Language Models with Preferences through f-divergence Minimization (2023, background): “Self-BLEU-n (Zhu et al., 2018), a measure of text diversity on a distributional level across samples.”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “Self-BLEU-n, a measure of text diversity on a distributional level across samples proposed by (Zhu et al., 2018), ensuring that policies don’t converge into limited number of sequences that satisfy the imposed constraints Caccia et al. (2020).”
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, methodology): “Following the intuition of self-BLEU (Zhu et al., 2018), we deﬁne soft uniqueness to describe diversity of generations in a corpus.”
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, methodology): “For a given x and its counterfactuals X̂, we approx- imate diversity using self-BLEU (Zhu et al., 2018) within X̂.”
LongForm: Effective Instruction Tuning with Reverse Instructions (2023, neutral): “We now present a comprehensive evaluation of our LongForm models compared to the baselines on LongForm-C test, recipe generation, long form question answering, short story generation, NLU, and ablation of LongForm-C subsets.”
Using In-Context Learning to Improve Dialogue Safety (2023, neutral): “B LEU metric, we randomly sample 128 responses from each model to compute the score.”
BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model (2019, methodology): “Diversity To measure the diversity of each model’s generations, we compute self-BLEU (Zhu et al., 2018): for each generated sentence, we compute BLEU treating the rest of the sentences as references, and average across sentences.”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “Text generation is typically evaluated along two axes: the quality (with metrics like mean perplexity or negative log likelihood) and the diversity (with metrics like selfBLEU2 (Zhu et al., 2018)) of the generated text.”
A Distributional Approach to Controlled Text Generation (2020, background): “So additionally, we report SelfBLEU-3,4,5 (Zhu et al., 2018) to measure repetitions at a distributional level across the whole set of generated samples, and also provide a token/type frequency analysis (see Fig.”
Hurdles to Progress in Long-form Question Answering (2021, neutral): “A possible ﬁx is a sentence-level evaluation and then aggregating scores across generated sentences, but appropriate penalties are needed for lack of diversity (Zhu et al., 2018) and short lengths.”
Language GANs Falling Short (2018, background): “As a remedy, Zhu et al. (2018) propose a metric that compares a generated sentence with a corpus of generated sentences, called Self-BLEU.”
Evaluation of Text Generation: A Survey (2020, background): “Many NLG surveys have been published in the last few years (Gatt & Krahmer, 2017; Zhu et al., 2018; Zhang et al., 2019a).”
Understanding the Effects of RLHF on LLM Generalisation and Diversity (2023, background): “Several works (Khalifa et al., 2021; Perez et al., 2022) have shown in specific settings that RLHF fine-tuning produces models with less output diversity, as measured by self-BLEU (Zhu et al., 2018).”
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (2023, methodology): “We quantify diversity using Self-BLEU (Zhu et al., 2018), where a score of 100 indicates deterministic outputs and 0 signifies maximum diversity.”
Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model (2023, methodology): “Concretely, we We compute the ratio of semantically equivalent pairs and their average Self-BLEU. first sample 1000 context (each with 1-5 sentences) from news articles in XSUM dataset (Narayan et al., 2018).”