Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems

By Weijie Zhao, Deping Xie, Ronglai Jia, Yulei Qian, Rui Ding, Mingming Sun, P. Li. 2020

161 citations recorded by the source corpus.

Neural networks of ads systems usually take input from multiple resources, e.g., query-ad relevance, ad features and user portraits. These inputs are encoded into one-hot or multi-hot binary features, with typically only a tiny fraction of nonzero feature values per example. Deep learning models in online advertising industries can have terabyte-scale parameters that do not fit in the GPU memory nor the CPU main memory on a computing node. For example, a sponsored online advertising system can contain more than $10^{11}$ sparse features, making the neural network a massive model with around 10 TB parameters. In this paper, we introduce a distributed GPU hierarchical parameter server for massive scale deep learning ads systems. We propose a hierarchical workflow that utilizes GPU High-Bandwidth Memory, CPU main memory and SSD as 3-layer hierarchical storage. All the neural network training computations are contained in GPUs. Extensive experiments on real-world data confirm the effectiveness and the scalability of the proposed system. A 4-node hierarchical GPU parameter server can train a model more than 2X faster than a 150-node in-memory distributed parameter server in an MPI cluster. In addition, the price-performance ratio of our proposed system is 4-9 times better than an MPI-cluster solution.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

In Optimization & Training

Citations in the corpus

ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep learning (2021, methodology): “[27] use a hierarchical parameter server-based design to offload sparse parameters to SSD for creating a massive scale DL Ads System.”
Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “Similarly, Figure 2(c) illustrates that between 2019 and 2021, the size of recommendation models at Facebook has increased by 20× [15], [16], [17], [11].”
RecSSD: near data processing for solid state drive based recommendation inference (2021, methodology): “In fact, training platforms for terabyte scale personalized recommendation models rely heavily on SSD storage capabilities for efficient and scalable execution [41].”
RecShard: statistical feature-based memory optimization for industry-scale neural recommendation (2022, methodology): “EMB memory footprints are on the order of terabytes (TB) [26, 46] and account for over 99% of the total model capacity [11].”
Understanding Capacity-Driven Scale-Out Neural Recommendation Inference (2020, methodology): “Notably, it is distinct from recent throughput-oriented training systems that do not have the same latency constraints found in inference serving [12], [15].”
Heterogeneous Attention Network for Effective and Efficient Cross-modal Retrieval (2021, methodology): “In particular, the work has been deployed for production, based on the “PaddleBox” training platform [51, 52].”
CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery (2020, background): “MLP layers are replicated across multiple nodes (trainers) and run in parallel, while Emb layers are sharded across multiple embedding parameter server nodes (Emb PS) due to their large size (Zhao et al., 2020).”
LIRA: Learnable, Imperceptible and Robust Backdoor Attacks (2021, background): “Machine learning models, especially deep neural networks (DNNs), have recently achieved state-of-the-art performance in various applications and tasks, ranging from conventional research topics such as computer vision [24, 22] and natural language processing [13, 16] to distant fields such as games ”
Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale (2021, background): “For instance, the memory requirement of deep learning recommender systems has grown from few tens of GBs to the terabyte regime [41, 63].”
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale (2020, methodology): “The memory capacity of embedding tables have increased dramatically from tens of GBs to TBs throughout the industry [36], [56], [58]–[60].”