EvalAI: Towards Better Evaluation Systems for AI Agents

By Deshraj Yadav, Rishabh Jain, Harsh Agrawal, Prithvijit Chattopadhyay, Taranjeet Singh, Akash Jain, Shivkaran Singh, Stefan Lee, Dhruv Batra. 2019

77 citations recorded by the source corpus.

We introduce EvalAI, an open source platform for evaluating and comparing machine learning (ML) and artificial intelligence algorithms (AI) at scale. EvalAI is built to provide a scalable solution to the research community to fulfill the critical need of evaluating machine learning models and agents acting in an environment against annotations or with a human-in-the-loop. This will help researchers, students, and data scientists to create, collaborate, and participate in AI challenges organized around the globe. By simplifying and standardizing the process of benchmarking these models, EvalAI seeks to lower the barrier to entry for participating in the global scientific effort to push the frontiers of machine learning and artificial intelligence, thereby increasing the rate of measurable progress in this domain.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

KILT: a Benchmark for Knowledge Intensive Language Tasks (2020, methodology): “The authors would like to greatly thank the team behind Natural Questions 14 for the held out data, that deﬁnes our NQ test set; FEVER 15 , HotpotQA and TriviaQA 17 teams for sharing ofﬁcial test data for the KILT leaderboard; Luke Zettlemoyer and Scott Wen-tau Yih for helpful discussions; Rishabh J”
nocaps: novel object captioning at scale (2019, methodology): “To facilitate evaluation and avoid exposing the novel object captions, we host an evaluation server for nocaps on EvalAI [45] – as such, we put forth these guidelines for using nocaps: –”
Hurdles to Progress in Long-form Question Answering (2021, positive): “We outperform prior work on both generation & combined scores. are hidden, and hosted on a public leaderboard in the EvalAI platform (Yadav et al., 2019).”
ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning (2020, methodology): “We are also hosting a public evaluation server on EvalAI (Yadav et al., 2019) to benchmark progress on Reclor.”
A Meta-Analysis of Overfitting in Machine Learning (2019, background): “Finally, another important question is whether other competition platforms such as CodaLab [1] or EvalAI [22] also show little signs of adaptive overﬁtting.”
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (2022, methodology): “10 [127] Deshraj Yadav, Rishabh Jain, Harsh Agrawal, Prithvijit Chattopadhyay, Taranjeet Singh, Akash Jain, Shiv Baran Singh, Stefan Lee, and Dhruv Batra.”
Overview and Insights from the Shared Tasks at Scholarly Document Processing 2020: CL-SciSumm, LaySumm and LongSumm (2020, methodology): “The evaluation was executed on a public leaderboard 11 , forked from EvalAI (Yadav et al., 2019), an open-source AI challenge hosting platform.”
DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue (2020, methodology): “The DialoGLUE leaderboard and evaluation scripts are hosted using the EvalAI platform (Yadav et al., 2019).”