Cut the CARP: Fishing for zero-shot story evaluation
By Shahbuland Matiana, J. Smith, Ryan Teehan, Louis Castricato, Stella Biderman, Leo Gao, Spencer J. Frazier. 2021
16 citations recorded by the source corpus.
Recent advances in large-scale language models (Raffel et al., 2019; Brown et al., 2020) have brought significant qualitative and quantitative improvements in machine-driven text generation. Despite this, generation and evaluation of machine-generated narrative text remains a challenging problem. Objective evaluation of computationally-generated stories may be prohibitively expensive, require meticulously annotated datasets, or may not adequately measure the logical coherence of a generated story's narratological structure. Informed by recent advances in contrastive learning (Radford et al., 2021), we present Contrastive Authoring and Reviewing Pairing (CARP): a scalable, efficient method for performing qualitatively superior, zero-shot evaluation of stories. We show a strong correlation between human evaluation of stories and those of CARP. Model outputs more significantly correlate with corresponding human input than those language-model based methods which utilize finetuning or prompt engineering approaches. We also present and analyze the Story-Critique Dataset, a new corpora composed of 1.3 million aligned story-critique pairs derived from over 80,000 stories. We expect this corpus to be of interest to NLP researchers.
Subfields: Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Natural Language Processing
- Mohit Iyyer
- John Wieting
- Kalpesh Krishna
- Angelica Chen
- Graham Neubig
- Ethan Perez
- Kyunghyun Cho
- Kevin Gimpel
In Evaluation & Benchmarks
- Ethan Perez
- Kyunghyun Cho
- Angelica Chen
- Yapei Chang
- Jon Ander Campos
- John Wieting
- J'er'emy Scheurer
- Shahbuland Matiana
Citations in the corpus
- Self-critiquing models for assisting human evaluators (2022, neutral): “[MST21] introduce a dataset of story critiques.”
- Training Language Models with Language Feedback (2022, background): “Matiana et al. (2021) learn text embeddings of language feedback, where improvements could benefit the refinement-scoring step of our algorithm.”
- RankGen: Improving Text Generation with Large Ranking Models (2022, methodology): “…or by fine-tuning on retrieval benchmarks like BEIR (Thakur et al., 2021) • utilizing of R ANK G EN as a text generation evaluation metric like CARP (Matiana et al., 2021) or CLIPScore (Hessel et al., 2021) • using RankGen on other domains with sequential data, like code completion, protein…”
- Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning (2022, methodology): “To circumvent this explicit data requirement while allowing a large class of preferences we leverage CARP (Matiana et al., 2021).”
- Training Language Models with Language Feedback at Scale (2023, background): “Matiana et al. (2021) learn text embeddings of language feedback, where improvements could benefit the refinement-scoring step of our algorithm.”