The Efficiency Misnomer

By Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, Yi Tay. 2021

121 citations recorded by the source corpus.

Model efficiency is a critical aspect of developing and deploying machine learning models. Inference time and latency directly affect the user experience, and some applications have hard requirements. In addition to inference costs, model training also have direct financial and environmental impacts. Although there are numerous well-established metrics (cost indicators) for measuring model efficiency, researchers and practitioners often assume that these metrics are correlated with each other and report only few of them. In this paper, we thoroughly discuss common cost indicators, their advantages and disadvantages, and how they can contradict each other. We demonstrate how incomplete reporting of cost indicators can lead to partial conclusions and a blurred or incomplete picture of the practical considerations of different models. We further present suggestions to improve reporting of efficiency metrics.

Subfields: Artificial Intelligence, Philosophy

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Philosophy

Citations in the corpus

UL2: Unifying Language Learning Paradigms (2022, background): “It is clear from these results that encoder-decoder models should be preferred over decoder-only models if and only if there is no concern about storage, i.e., parameter counts are generally less important than actual throughput (see (Dehghani et al., 2021a) for a detailed discussion).”
Fast Inference from Transformers via Speculative Decoding (2022, background): “The efﬁciency of inference from large models was studied extensively (Dehghani et al., 2021).”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, background): “We note that FLOPs are not a direct measurement of real-world computational cost because latency, power usage, and other costs can vary significantly depending on hardware and other factors [52].”
LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, methodology): “1, 2, 4 [13] Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay.”
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers (2021, background): “Parameter count is reflective neither of speed, nor of capacity (10).”
Efficient Transformers: A Survey (2020, background): “Eﬃciency and computational costs is generally a complicated aﬀair and we would suggest readers peruse (Dehghani et al., 2021) for more details on trade-oﬀs, intricacies etc.”
DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale (2022, background): “For example, all micro-operations in a layer-norm [33] can be tiled along the token dimension, while the reduction dimensions are processed within a tile.”
Transcending Scaling Laws with 0.1% Extra Compute (2022, neutral): “Figure 2: Computation cost (training flops) (Dehghani et al., 2021) versus Quality (average of 20+ NLP zero and few-shot tasks).”
Scaling Vision Transformers to 22 Billion Parameters (2023, background): “ViT-22B’s model ﬂops utilization (MFU) (Chowdhery et al., 2022; Dehghani et al., 2021a) is 54.9% , indicating a very efﬁcient use of the hardware.”
Confident Adaptive Language Modeling (2022, background): “Our main efficiency metric is the average number of decoder layers used per output token, as it directly measures complexity reduction without conflating with implementation or infrastructure specific details [19].”
Unifying Language Learning Paradigms (2022, background): “We also highlight the fact that the overall gain is normalized , so this becomes harder to exploit or be susceptible to benchmark lottery effects (Dehghani et al., 2021b).”
SCENIC: A JAX Library for Computer Vision Research and Beyond (2021, methodology): “On top of that, SCENIC has been used in numerous Google projects and research papers such as ViViT [6], OmniNet [28], TokenLearner [25], MBT [21], PolyViT [19], MTV [31], studies on scaling behaviour [5, 10, 29] and efficiency [9] of various models among others [15–18, 20].”
HyperPrompt: Prompt-based Task-Conditioning of Transformers (2022, background): “, prompts/adapters) simply presents an illusion of efficiency (Dehghani et al., 2021).”
Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better (2021, methodology): “We encourage the reader to refer to the work of Dehghani [37] for a detailed treatment on this subject.”
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? (2022, background): “Hence, compared with standard Transformers, this architectural choice significantly warps the scaling behaviour not only with respect to performance but also amongst compute metrics such as FLOPs, speed and number of parameters (Dehghani et al., 2021a).”
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints (2022, methodology): “…vs. cost plots, where we use the upstream or downstream performance to measure quality, and training time in terms of TPU-core-days (as prominent cost metrics (Dehghani et al., 2021)) or training steps (when the cost per step is the same for all the compared models) to measure computation cost.”
MobileOne: An Improved One millisecond Mobile Backbone (2022, methodology): “However, these metrics may not correlate well with the efficiency [9] of the models in terms of latency.”
EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention (2023, background): “However, most of them target at minimizing Flops and parameters [16], which could have low correlations with actual inference latency [70] and still inferior to efficient CNNs in speed.”