Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement

By JosÃ© Hernandez-Orallo. 2017

188 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Unmasking Clever Hans predictors and assessing what machines really learn (2019, methodology): “An overview of the FV and DNN strategies for the other classes and for the Atari Pinball and Breakout game can be found in Supplementary Figures 23 – 25 and 30 – 32, respectively.”
Human-Centered AI (2021, methodology): “Defining standard methods and processes for evaluating system outcomes is a significant need to advance the discipline [28] .”
The Measure of Intelligence (2019, unknown): “This definition and evaluation philosophy focused entirely on skill at narrow tasks normally handled by humans has led to a striking 2Note the lingering influence of the Turing Test. paradox, as pointed out by Hernández-Orallo [36] in his 2017 survey: the field of artificial intelligence has been v”
Measuring the Occupational Impact of AI: Tasks, Cognitive Abilities and AI Benchmarks (2020, methodology): “When aiming at evaluating the progress in a speciﬁc (AI) discipline, we need to focus on objective evaluation tools to measure the elements and objects of study, assess the prototypes and artefacts that are being built and examine the discipline as a whole (Hernández-Orallo, 2017b).”
α-Rank: Multi-Agent Evaluation by Evolution (2019, background): “Researchers have also carried out studies of evaluation metrics in the fields of computer science, machine learning, and artificial intelligence [90–92].”
Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications (2021, result): “The results of our probes further offer evidence towards the growing body of work [12] [11] pointing to the insufficiency of assessing models only on task-oriented capability evaluation metrics — without taking into consideration different deployment contexts and different people who would interact ”
What Makes Reading Comprehension Questions Easier? (2018, background): “Evaluation overﬁtting: The theory behind evaluating AI distinguishes between task-and skill-oriented approaches (Hern´andez-Orallo, 2017).”