Scarecrow: A Framework for Scrutinizing Machine Text

By Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, Yejin Choi. 2021

36 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “…Zellers et al., 2019), summarization (Gabriel et al., 2021; Maynez et al., 2020; Stiennon et al., 2020; Wang et al., 2020), conversational dialog (Shuster et al., 2021; Roller et al., 2021), and question answering (Dou et al., 2021; Krishna et al., 2021; Lewis et al., 2020; Logan IV et al., 2019).”
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, methodology): “, 2020), which is known to produce closeto-human-like text (Clark et al., 2021; Dou et al., 2021) but also easily generates socially biased and toxic content (Sheng et al.”
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022, methodology): “Despite known deficiencies such as incoherence or repetition (Dou et al., 2021), these models often produce human-like text (Clark”
The Gradient of Generative AI Release: Methods and Considerations (2023, methodology): “Approaches to detection can be tailored to modality [42] such as text [35], and include human annotation [29].”
CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022, neutral): “[20]’s fnding that sentences written by writers tend to have more grammar and usage errors compared to those written by GPT-3.”
How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN (2021, unknown): “, 2020a) or via scores that focus on specific properties (Dou et al., 2021) such as fluency (Mutton et al.”
Cut the CARP: Fishing for zero-shot story evaluation (2021, background): “Recently, researchers interested in story evaluation have developed more sophisticated techniques for evaluating stories (Akoury et al., 2020; Dou et al., 2021) as well as the Purdy Index (Purdy et al., 2018) and the Fabula-Entropy Index (Castricato et al., 2021a;b).”
Underreporting of errors in NLG output, and what to do about it (2021, background): “Alternatively, Dou et al. (2021) present a crowd-sourced error annotation schema called SCARECROW .”
Do Language Models Plagiarize? (2022, background): “Varying effects of decoding methods and their parameters on text quality and diversity have been extensively studied (DeLucia et al., 2020; Dou et al., 2021; Basu et al., 2020), but not from the plagiarism perspective.”
PERSONACHATGEN: Generating Personalized Dialogues using GPT-3 (2022, background): “…as an alternate way, many studies have created datasets by leveraging pre-trained language models with designed prompt instructions (Yoo et al., 2021; Baheti et al., 2021; Hartvigsen et al., 2022) due to their enormous ability to produce more human-like text (Clark et al., 2021; Dou et al., 2021).”