Efficiency & Compression

Quantization, distillation, pruning, parameter-efficient fine-tuning (LoRA/adapters), efficient inference.

A topic in the Science corpus, drawn from 918 papers.

Trusted voices in Efficiency & Compression

Ordered by the trust Efficiency & Compression's own network concentrates on each researcher, most first.

  1. Geoffrey E. Hinton — trusted by 1,356 researchers here
  2. Oriol Vinyals — trusted by 1,194 researchers here
  3. Joseph Dean — trusted by 1,179 researchers here
  4. Song Han — trusted by 901 researchers here
  5. Quoc V. Le — trusted by 833 researchers here
  6. William (Bill) J. Dally — trusted by 811 researchers here
  7. Mingxing Tan — trusted by 802 researchers here
  8. Xiangyu Zhang — trusted by 589 researchers here
  9. Yoshua Bengio — trusted by 489 researchers here
  10. Huizi Mao — trusted by 472 researchers here
  11. Matthieu Courbariaux — trusted by 393 researchers here
  12. Andrew G. Howard — trusted by 421 researchers here

Most-cited papers in Efficiency & Compression

Ordered by the recorded citation count from the source corpus; a citation is not an endorsement.

  1. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications (2017) — 25,507 citations
  2. MobileNetV2: Inverted Residuals and Linear Bottlenecks (2018) — 25,197 citations
  3. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019) — 25,169 citations
  4. Distilling the Knowledge in a Neural Network (2015, arXiv.org) — 25,160 citations
  5. LoRA: Low-Rank Adaptation of Large Language Models (2021, International Conference on Learning Representations) — 21,145 citations
  6. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2015) — 10,226 citations
  7. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019) — 10,143 citations
  8. Searching for MobileNetV3 (2019) — 9,744 citations
  9. Training data-efficient image transformers & distillation through attention (2020) — 9,450 citations
  10. ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices (2017) — 8,533 citations
  11. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size (2016) — 8,474 citations
  12. Learning both Weights and Connections for Efficient Neural Network (2015) — 7,826 citations
  13. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019) — 7,601 citations
  14. Enhanced Deep Residual Networks for Single Image Super-Resolution (2017) — 7,355 citations
  15. EfficientDet: Scalable and Efficient Object Detection (2019) — 7,168 citations
  16. Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) — 7,142 citations
  17. Parameter-Efficient Transfer Learning for NLP (2019) — 6,877 citations
  18. ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design (2018) — 6,472 citations
  19. ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks (2019) — 6,251 citations
  20. Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021) — 6,083 citations
  21. Federated Learning: Strategies for Improving Communication Efficiency (2016) — 5,508 citations
  22. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017) — 5,113 citations
  23. QLoRA: Efficient Finetuning of Quantized LLMs (2023, Neural Information Processing Systems) — 5,083 citations
  24. Bag of Tricks for Efficient Text Classification (2016) — 5,082 citations
  25. Coordinate Attention for Efficient Mobile Network Design (2021) — 5,022 citations
  26. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, Neural Information Processing Systems) — 4,798 citations
  27. FitNets: Hints for Thin Deep Nets (2014) — 4,770 citations
  28. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016) — 4,754 citations
  29. EfficientNetV2: Smaller Models and Faster Training (2021) — 4,600 citations
  30. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference (2017) — 4,475 citations
  31. Knowledge Distillation: A Survey (2020) — 4,395 citations
  32. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018) — 4,345 citations
  33. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021, Journal of machine learning research) — 4,336 citations
  34. Pruning Filters for Efficient ConvNets (2016) — 4,125 citations
  35. MnasNet: Platform-Aware Neural Architecture Search for Mobile (2018) — 3,454 citations
  36. Accelerating the Super-Resolution Convolutional Neural Network (2016) — 3,453 citations
  37. Towards Federated Learning at Scale: System Design (2019) — 3,255 citations
  38. BinaryConnect: Training Deep Neural Networks with binary weights during propagations (2015) — 3,245 citations
  39. Reformer: The Efficient Transformer (2020) — 3,006 citations
  40. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019) — 2,986 citations
  41. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (2020) — 2,953 citations
  42. Model compression (2006) — 2,875 citations
  43. Learning Efficient Convolutional Networks through Network Slimming (2017) — 2,811 citations
  44. Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks (2016) — 2,806 citations
  45. Channel Pruning for Accelerating Very Deep Neural Networks (2017) — 2,781 citations
  46. Regularization of Neural Networks using DropConnect (2013) — 2,649 citations
  47. Active Learning for Convolutional Neural Networks: A Core-Set Approach (2017) — 2,623 citations
  48. EIE: Efficient Inference Engine on Compressed Deep Neural Network (2016) — 2,611 citations
  49. Learning Structured Sparsity in Deep Neural Networks (2016) — 2,511 citations
  50. Vector Quantization (2017) — 2,461 citations

Most active researchers

By papers in this topic: a volume measure, not trust.