Challenges in Deploying Machine Learning: A Survey of Case Studies

By Andrei Paleyes, Raoul-Gabriel Urma, Neil D. Lawrence. 2020

636 citations recorded by the source corpus.

In recent years, machine learning has transitioned from a field of academic research interest to a field capable of solving real-world business problems. However, the deployment of machine learning models in production systems can present a number of issues and concerns. This survey reviews published reports of deploying machine learning solutions in a variety of use cases, industries, and applications and extracts practical considerations corresponding to stages of the machine learning deployment workflow. By mapping found challenges to the steps of the machine learning deployment workflow, we show that practitioners face issues at each stage of the deployment process. The goal of this article is to lay out a research agenda to explore approaches addressing these challenges.

Subfields: Artificial Intelligence, ML Systems & Hardware, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yi Tay
2. Furu Wei
3. Mostafa Dehghani
4. Julian McAuley
5. Anurag Arnab
6. Canwen Xu
7. Jianfeng Gao
8. Tao Ge

In ML Systems & Hardware

1. Anurag Arnab
2. Mostafa Dehghani
3. Yi Tay
4. Julian McAuley
5. Lucas Beyer
6. Ashish Vaswani

In Human-Computer Interaction

1. Anurag Arnab
2. Mostafa Dehghani

Citations in the corpus

• The Efficiency Misnomer (2021, background): “One of the main considerations in designing neural network architectures is quality-cost trade-off (Paleyes et al., 2020).”
• Addressing "Documentation Debt" in Machine Learning: A Retrospective Datasheet for BookCorpus (2021, background): “” Although prominent research tends to be model-focused, multiple reviews [34, 37] suggest that data management creates many challenges and takes up a substantial portion of many ML workflows, including dataset generation, preprocessing, and augmentation.”
• Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression (2021, background): “In real-world settings, training and deployment are often separate (Paleyes et al., 2020).”