Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better

By Gaurav Menghani. 2021

652 citations recorded by the source corpus.

Deep learning has revolutionized the fields of computer vision, natural language understanding, speech recognition, information retrieval, and more. However, with the progressive improvements in deep learning models, their number of parameters, latency, and resources required to train, among others, have all increased significantly. Consequently, it has become important to pay attention to these footprint metrics of a model as well, not just its quality. We present and motivate the problem of efficiency in deep learning, followed by a thorough survey of the five core areas of model efficiency (spanning modeling techniques, infrastructure, and hardware) and the seminal work there. We also present an experiment-based guide along with code for practitioners to optimize their model training and deployment. We believe this is the first comprehensive survey in the efficient deep learning space that covers the landscape of model efficiency from modeling techniques to hardware support. It is our hope that this survey would provide readers with the mental model and the necessary understanding of the field to apply generic efficiency techniques to immediately get significant improvements, and also equip them with ideas for further research and experimentation to achieve additional gains.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

The Efficiency Misnomer (2021, methodology): “Aside from model quality, the efﬁciency (Menghani, 2021) of a model is often an important aspect to consider and is commonly used to measure the relative utility of different methods.”
Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing (2024, background): “As such, we expect inference cost to dominate the overall cost of such models and hence focus on works that reduce the cost of ML inference (Menghani, 2023).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “compression methods like pruning, quantization, knowledge distillation, parameter sharing, and factorization are explored to develop compact models in English for general domain applications [162], [290].”
Recall Before Rerank: Benchmarking Deep Learning Models for Large-Scale Code-to-Code Retrieval (2026, unknown): “Our primary research questions are as follows: RQ For RQ 2 , we show in Section 6 that embedding rate is scale-critical (e.g., Big-CloneBench’s 6M methods [32]), making IR methods or lossy embedding compression [19,25][7, §13.7] more practical for large corpora.”