Understanding Capacity-Driven Scale-Out Neural Recommendation Inference

By Michael Lui, Yavuz Yetim, Özgür Özkan, Zhuoran Zhao, Shin-Yeh Tsai, Carole-Jean Wu, Mark Hempstead. 2020

58 citations recorded by the source corpus.

Deep learning recommendation models have grown to the terabyte scale. Traditional serving schemes–that load entire models to a single server–are unable to support this scale. One approach to support these models is distributed serving, or distributed inference, which divides the memory requirements of a single large model across multiple servers. This work is a first-step for the systems community to develop novel model-serving solutions, given the huge system design space. Large-scale deep recommender systems are a novel workload and vital to study, as they consume up to 79% of all inference cycles in the data center. To that end, this work is the first to describe and characterize scale-out deep learning recommender inference using data-center serving infrastructure. This work specifically explores latency-bounded inference systems, compared to the throughput-oriented training systems of other recent works. We find that the latency and compute overheads of distributed inference are largely attributed to a model's static embedding table distribution and sparsity of inference request inputs. We evaluate three embedding table mapping strategies on three representative models and specify the challenging design trade-offs in terms of end-to-end latency, compute overhead, and resource efficiency. Overall, we observe a modest latency overhead with distributed inference–P99 latency is increased by only 1% in the best case configuration. The latency overheads are a result of the commodity infrastructure used and the sparsity of embedding tables. Encouragingly, we also show how distributed inference can account for efficiency improvements in data-center scale recommendation serving.

Subfields: Artificial Intelligence, Information Systems, Recommender Systems, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Recommender Systems

In ML Systems & Hardware

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “Similarly, Figure 2(c) illustrates that between 2019 and 2021, the size of recommendation models at Facebook has increased by 20× [15], [16], [17], [11].”
RecSSD: near data processing for solid state drive based recommendation inference (2021, background): “For instance, compared to other AI-driven applications, recommendation accounts for 10× the infrastructure capacity in Facebook’s datacenter [20, 28, 30].”
RecShard: statistical feature-based memory optimization for industry-scale neural recommendation (2022, methodology): “This is due to their use of embedding layers which demand ordersof-magnitude higher memory capacity and exhibit significantly lower compute-intensity [11, 26, 33].”
Understanding and Co-designing the Data Ingestion Pipeline for Industry-Scale RecSys Training (2021, background): “Secondly, training these models, which often consist of trillions of parameters [32, 37], places enormous demands on the end-to-end training and data ingestion pipeline.”
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale (2020, background): “The memory capacity of embedding tables have increased dramatically from tens of GBs to TBs throughout the industry [36], [56], [58]–[60].”
TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models (2021, background): “Furthermore, the memory capacity requirement for state-of-the-art recommendation models has grown into the scale of terabytes (Lui et al., 2021; Yi et al., 2018; Zhao et al., 2019; 2020).”
CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery (2020, methodology): “As the size and the complexity of recommendation models grow (Zhao et al., 2020; Lui et al., 2021), the scale of MLP trainers and Emb PS nodes increases quickly, that leads to growing failure rates of training jobs.”
Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale (2021, background): “For instance, the memory requirement of deep learning recommender systems has grown from few tens of GBs to the terabyte regime [41, 63].”