The ML test score: A rubric for ML production readiness and technical debt reduction

By Eric Breck, Shanqing Cai, Eric Nielsen, M. Salib, David Sculley. 2017 IEEE International Conference on Big Data (Big Data), 2017

301 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/d15d3dea2982a670863c879b4fb4f5183d10aca1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Ben Hutchinson
2. Emily L. Denton
3. Inioluwa Deborah Raji
4. A. Hanna
5. Vinodkumar Prabhakaran
6. Negar Rostamzadeh
7. Yang Liu
8. Margaret Mitchell

Citations in the corpus

• Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing (2020, background): “rformance metrics such as accuracy or profit [39]. Although concerns about reliability are related, and although practices for testing production AI systems are established for industry practitioners [4], issues involving social impact, downstream effects in critical domains, and ethics and fairness”
• Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure (2020, background): “When requirements and their testing is specified clearly enough, opportunities for test automation become possible, including realtime monitoring and alerting [23].”
• Machine Learning Testing: Survey, Landscapes and Horizons (2019, methodology): “[154] offered 28 test aspects to consider and a scoring system used by Google.”
• Software Engineering for AI-Based Systems: A Survey (2021, methodology): “These are the corresponding Knowledge Areas: Software Requirements, Software Design, Software Construction, Software Testing, Software Maintenance, Software Configuration Management, ACM Transactions on Software Engineering and Methodology, Vol. 31, No. 2, Article 37e. Publication date: March 2022.”
• Automating Large-Scale Data Quality Verification (2018, background): “…involved in building complex machine learning applications in the real-world have recently been highlighted by many researchers, e.g., with respect to managing the resulting models [26], software engineering [42, 8], pipeline abstractions [2, 43, 46], and learnings from real-world systems [34, 7].”
• Evaluation Gaps in Machine Learning Practice (2022, background): “…information needs of application developers, we are also aligned with calls for transparent reporting of ML model evaluations [117], prioritizing needs of ML fairness practitioners [77], model auditing practices [136], and robust practices for evaluating ML systems for production readiness [23].”
• Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare (2022, background): “Given the complexity of ML algorithms, a number of papers have suggested monitoring ML explainability metrics, such as variable importance (VI) 18,24 .”