Evaluating Machines by their Real-World Language Use

By Rowan Zellers, Ari Holtzman, Elizabeth Clark, Lianhui Qin, Ali Farhadi, Yejin Choi. 2020

13 citations recorded by the source corpus.

There is a fundamental gap between how humans understand and use language -- in open-ended, real-world situations -- and today's NLP benchmarks for language understanding. To narrow this gap, we propose to evaluate machines by their success at real-world language use -- which greatly expands the scope of language tasks that can be measured and studied. We introduce TuringAdvice, a new challenge for language understanding systems. Given a complex situation faced by a real person, a machine must generate helpful advice. We make our challenge concrete by introducing RedditAdvice, a dataset and leaderboard for measuring progress. Though we release a training set with 600k examples, our evaluation is dynamic, continually evolving with the language people use: models must generate helpful advice for recently-written situations. Empirical results show that today's models struggle at our task, even those with billions of parameters. The best model, a finetuned T5, writes advice that is at least as helpful as human-written advice in only 9% of cases. This low performance reveals language understanding errors that are hard to spot outside of a generative setting, showing much room for progress.

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Experience Grounds Language (2020, background): “More flexible, dynamic evaluation (Zellers et al., 2020; Dinan et al., 2019) are a partial solution, but true persistence of identity and adaption to change are both necessary and still a long way off.”
Measuring Massive Multitask Language Understanding (2020, background): “However, NLG is notoriously difﬁcult to evaluate and lacks a standard metric (Sai et al., 2020).”
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021, background): “Many task-speciﬁc papers contribute to this debate, as does a prominent recent thread advocating for an increased focus on grounding of various kinds by Bender and Koller (2020), Bisk et al. (2020), Zellers et al. (2020), and others.”
Evaluation of Text Generation: A Survey (2020, methodology): “, 2019), and TuringAdvice (Zellers et al., 2020) tests models’ language understanding capabilities by having people read and rate the models’ ability to generate advice.”
G ENIE A Leaderboard for Human-in-the-Loop Evaluation of Text Generation (2021, background): “For example, HYPE (Zhou et al., 2019) is an evaluation platform for image generation, ChatEval (Sedoc et al., 2019) is an evaluation platform for chatbots, and more recently, Zellers et al. (2020) present a leaderboard for the advice generation task introduced in their work.”
Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models (2021, background): “By having writers interact with dynamically generated suggestions, collaborative writing systems reward helpful and robust generation models, underemphasized attributes in current evaluations (Zellers et al., 2021; Ethayarajh and Jurafsky, 2020).”