Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications

By Jongsoo Park, Maxim Naumov, Protonu Basu, Summer Deng, Aravind Kalaiah, Daya Shanker Khudia, James Law, Parth Malani, A. Malevich, Nadathur Rajagopalan Satish and 18 more. 2018

205 citations recorded by the source corpus.

The application of deep learning techniques resulted in remarkable improvement of machine learning models. In this paper provides detailed characterizations of deep learning models used in many Facebook social network services. We present computational characteristics of our models, describe high performance optimizations targeting existing systems, point out their limitations and make suggestions for the future general-purpose/accelerated inference hardware. Also, we highlight the need for better co-design of algorithms, numerics and computing platforms to address the challenges of workloads often run in data centers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Information Systems, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Information Systems

In Optimization & Training

Citations in the corpus

Chasing Carbon: The Elusive Environmental Footprint of Computing (2020, background): “In less than two years, Facebook hardware devoted to AI training and inference has grown by 4 × and 3.5 × , respectively [22], [23].”
Differentiable Neural Input Search for Recommender Systems (2020, background): “Another notable fact is that embedding layers in industrial web-scale recommender systems [10, 31] account for the majority of model parameters and can consume hundreds of gigabytes of memory space.”
Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems (2020, background): “Each node processes the entire mini-batch of data and communicates the activations forward or error gradients backwards to other nodes.”
The Architectural Implications of Facebook's DNN-Based Personalized Recommendation (2019, methodology): “Based on the use case, the SLA requirements can vary from tens to hundreds of milliseconds [15, 36, 43].”
Software-hardware co-design for fast and scalable training of deep learning recommendation models (2021, methodology): “At Meta, we have been using recommendation models extensively for ranking and click through rate (CTR) prediction, including news feed and search services [15, 17, 42, 47].”
RecSSD: near data processing for solid state drive based recommendation inference (2021, background): “While large embedding tables enable higher personalization, they consume up to hundreds of GBs of storage [20, 32].”
Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems (2019, background): “At present, the out-sized memory consumption of standard embedding layers in these recommendation systems is a major burden on the memory hierarchy – an embedding layer for a single recommendation engine can consume tens of gigabytes of space [41, 44].”
NIPQ: Noise Injection Pseudo Quantization for Automated DNN Optimization (2022, background): “Although the accuracy or output quality has been used as a decisive metric to evaluate the superiority of neural networks, the computation cost and memory footprint are also critical measures to determine their practical utility in reality [1, 2].”
Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale (2021, methodology): “…which we now elaborate: Initial Characterization: We examine the speciﬁc operators employed by the given model and arrive at basic theoretical performance characteristics such as memory intensity or foot-print and compute intensity using standard techniques such as rooﬂine analysis [11][48].”
DeepStore: In-Storage Acceleration for Intelligent Queries (2019, background): “Recent work [41, 76] has explored reorganizing feature vectors in-storage for efficient search operations.”
MASR: A Modular Accelerator for Sparse RNNs (2019, background): “deployed across a wide set of applications and platforms [15], [17], [18], [19].”
SIGMA: A Sparse and Irregular GEMM Accelerator with Flexible Interconnects for DNN Training (2020, background): “While GEMMs continue to remain the favorite abstraction to unroll tensors to during the forward and backward passes, architects need to be cognizant of three trends: • Many GEMMs have irregular (or non-square) dimensions arising from minibatches and weight factorization [34].”
High-performance, Distributed Training of Large-scale Deep Learning Recommendation Models (2021, background): “At Facebookwe recommendation models extensively for ranking and click through rate (CTR) prediction, including News Feed and search services [12, 13, 38, 41], and is the single largest AI application in terms of infrastructure demand in its data-centers.”