RoFT: A Tool for Evaluating Human Detection of Machine-Generated Text
By Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Chris Callison-Burch. 2020
71 citations recorded by the source corpus.
In recent years, large neural networks for natural language generation (NLG) have made leaps and bounds in their ability to generate fluent text. However, the tasks of evaluating quality differences between NLG systems and understanding how humans perceive the generated text remain both crucial and difficult. In this system demonstration, we present Real or Fake Text (RoFT), a website that tackles both of these challenges by inviting users to try their hand at detecting machine-generated text in a variety of domains. We introduce a novel evaluation task based on detecting the boundary at which a text passage that starts off human-written transitions to being machine-generated. We show preliminary results of using RoFT to evaluate detection of machine-generated news articles.
Subfields: Artificial Intelligence, Sociology and Political Science, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yejin Choi
- Yulia Tsvetkov
- Noah A. Smith
- Sachin Kumar
- Mohit Bansal
- Elizabeth Clark
- Antonios Anastasopoulos
- Vidhisha Balachandran
In Sociology and Political Science
- Yulia Tsvetkov
- Yejin Choi
- Noah A. Smith
- Elizabeth Clark
- Maxwell Forbes
- Suchin Gururangan
- Muhammad Abdul-Mageed
- Laks V. S. Lakshmanan
In Safety, Robustness & Interpretability
- Yulia Tsvetkov
- Sachin Kumar
- Noah A. Smith
- Yejin Choi
- Vidhisha Balachandran
- Suchin Gururangan
- Lucille Njoo
- Antonios Anastasopoulos
Citations in the corpus
- Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “, 2021) Disinformation Machine-generated text detection (Dugan et al., 2020; Gehrmann et al., 2019)”
- All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021, background): “, 2021) asks evaluators to rate NLG models by their ability to generate helpful advice, and RoFT (Dugan et al., 2020) engages evaluators through a guessing game to determine the boundary between human- and machine-generated text.”
- Hurdles to Progress in Long-form Question Answering (2021, neutral): “One effort in this direction is Dugan et al. (2020), who reveal one generated sentence at a time and estimate system quality based on the number of sentences which fooled humans.”
- Automatic Detection of Machine Generated Text: A Critical Survey (2020, methodology): “Real or Fake Text (RoFT) tool : The RoFT tool (Dugan et al., 2020) focuses on evaluating human detection of text generated by TGM by asking humans to detect the sentence boundary at which the text transitions from human written text to machine generated text.”
- Scarecrow: A Framework for Scrutinizing Machine Text (2021, background): “In the RoFT system (Dugan et al., 2020), annotators attempt to detect the boundary between human- and machine-written text as a proxy for assessing quality.”