Swayam: distributed autoscaling to meet SLAs of machine learning inference services with resource efficiency

By Arpan Gujarati, Sameh Elnikety, Yuxiong He, Kathryn S. McKinley, Björn B. Brandenburg. 2017

131 citations recorded by the source corpus.

Developers use Machine Learning (ML) platforms to train ML models and then deploy these ML models as web services for inference (prediction). A key challenge for platform providers is to guarantee response-time Service Level Agreements (SLAs) for inference workloads while maximizing resource efficiency. Swayam is a fully distributed autoscaling framework that exploits characteristics of production ML inference workloads to deliver on the dual challenge of resource efficiency and SLA compliance. Our key contributions are (1) model-based autoscaling that takes into account SLAs and ML inference workload characteristics, (2) a distributed protocol that uses partial load information and prediction at frontends to provision new service instances, and (3) a backend self-decommissioning protocol for service instances. We evaluate Swayam on 15 popular services that were hosted on a production ML-as-a-service platform, for the following service-specific SLAs: for each service, at least 99% of requests must complete within the response-time threshold. Compared to a clairvoyant autoscaler that always satisfies the SLAs (i.e., even if there is a burst in the request rates), Swayam decreases resource utilization by up to 27%, while meeting the service-specific SLAs over 96% of the time during a three hour window. Microsoft Azure's Swayam-based framework was deployed in 2016 and has hosted over 100,000 services.

Subfields: Electrical and Electronic Engineering, Hardware and Architecture, Information Systems, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

In Hardware and Architecture

In Information Systems

In ML Systems & Hardware

Citations in the corpus

Chasing Carbon: The Elusive Environmental Footprint of Computing (2020, background): “Furthermore, recent work proposes scheduling batch-processing workloads during periods when renewable energy is readily available [81]–[85].”
MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving (2019, methodology): “Also, many ML models, especially deep learning, have deterministic inference time [41, 74]—they take fixed-size input vectors and have input-independent control flows.”