Fathom: reference workloads for modern deep learning methods

By Robert Adolf, Saketh Rama, Brandon Reagen, Gu-Yeon Wei, David M. Brooks. 2016

189 citations recorded by the source corpus.

Deep learning has been popularized by its recent successes on challenging artificial intelligence problems. One of the reasons for its dominance is also an ongoing challenge: the need for immense amounts of computational power. Hardware architects have responded by proposing a wide array of promising ideas, but to date, the majority of the work has focused on specific algorithms in somewhat narrow application domains. While their specificity does not diminish these approaches, there is a clear need for more flexible solutions. We believe the first step is to examine the characteristics of cutting edge models from across the deep learning community. Consequently, we have assembled Fathom: a collection of eight archetypal deep learning workloads for study. Each of these models comes from a seminal work in the deep learning community, ranging from the familiar deep convolutional neural network of Krizhevsky et al., to the more exotic memory networks from Facebook's AI research group. Fathom has been released online, and this paper focuses on understanding the fundamental performance characteristics of each model. We use a set of application-level modeling tools built around the TensorFlow deep learning framework in order to analyze the behavior of the Fathom workloads. We present a breakdown of where time is spent, the similarities between the performance profiles of our models, an analysis of behavior in inference and training, and the effects of parallelism on scaling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Evaluation & Benchmarks

Citations in the corpus

In-datacenter performance analysis of a tensor processing unit (2017, background): “Fathom is a promising new attempt at such a benchmark suite [3].”
MLPerf Training Benchmark (2019, methodology): “Fathom and Google TF Benchmarks (Adolf et al., 2016; Google, 2017) provide a reference suite of DNN models that span a wide application space, but they speciﬁcally measure model throughput and fail to account for accuracy.”
DAWNBench : An End-to-End Deep Learning Benchmark and Competition (2017, background): “Most existing benchmarks for deep learning performance [16, 10, 5, 2, 41, 8, 3] only measure proxy metrics such as the time to process one minibatch of data.”
MLPerf Inference Benchmark (2019, methodology): “Fathom [2] provides a suite of models that incorporate several layer types (e.g., convolution, fully connected, and RNN).”
Predicting the Computational Cost of Deep Learning Models (2018, result): “Work has been done which shows that one can predict the execution time for a new model by attempting to compare the model to similar models which have known performance [8], however, this can only yield very coarse estimates.”
DeepRecSys: A System for Optimizing End-To-End At-Scale Neural Recommendation Inference (2020, background): “Currently-available benchmarks for DNNs primarily focus on FC, CNNs, and RNNs [35], [45]– [47].”
RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing (2019, methodology): “Compared to CNNs, RNNs, and FCs [12], [69], [72], [82], the analysis demonstrates how recommendation models have unique storage,”
MASR: A Modular Accelerator for Sparse RNNs (2019, background): “Recently, RNNs have become the standard end-to-end deep learning approach for ASR [3], [4].”
The Architectural Implications of Facebook's DNN-Based Personalized Recommendation (2019, methodology): “DNN performance analysis and optimization Current publicly available benchmarks [3, 8, 19, 55] for DNNs focus on neural networks with FC, CNN, and RNN layers only.”
Benchmarking TPU, GPU, and CPU Platforms for Deep Learning (2019, background): “There are two types, real-world benchmark suites such as MLPerf [41], Fathom [3], BenchNN [12], and BenchIP [51], and micro-benchmark suites, such as Deep-Bench [43] and BenchIP.”
DeepOBS: A Deep Learning Optimizer Benchmark Suite (2019, background): “Fathom is another hardware-centric benchmark, which among other things assesses how computational resources are spent (Adolf et al., 2016).”
Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark (2018, background): “Furthermore, several benchmarks, including Fathom, do not benchmark distributed DL training [6,12,21].”
Google Workloads for Consumer Devices: Mitigating Data Movement Bottlenecks (2018, background): “D convolution layers and fully-connected layers [2], the majority of energy is spent on these two types of layers.”