The hardware lottery

By Sara Hooker. 2020

287 citations recorded by the source corpus.

After decades of incentivizing the isolation of hardware, software, and algorithm development, the catalysts for closer collaboration are changing the paradigm.

Subfields: Computational Theory and Mathematics, Computer Science Applications, Information Systems and Management, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Computer Science Applications

In Information Systems and Management

In ML Systems & Hardware

Citations in the corpus

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “However, even though structured matrices are efficient in theory, they have not seen wide adoption since it is hard to translate their efficiency to wall-clock speedup since dense unconstrained matrix multiply has very optimize implementation, a phenomenon known as the hardware lottery [39].”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “These include the particular scheme used for initialization; conventions for parameterization; choice of optimization algorithm; conventions for representing data; compression schemes; and choices of batch size, learning rate, and other hyperparameters, all of which may interact with the infrastruct”
High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “It is possible that future accelerators may be able to take full advantage of the potential training speed that largely goes unrealized with models like EfficientNets, so we believe this direction should not be ignored (Hooker, 2020), however we anticipate that developing models with improved traini”
The Efficiency Misnomer (2021, background): “A ﬁnal concern is that, methods which seemingly train faster tend to be used more often and hence get optimized more over time, with the danger of getting stuck in a local minimum (Hooker, 2020; Dehghani et al., 2021b).”
Long Range Arena: A Benchmark for Efficient Transformers (2020, background): “To succeed, a Transformer model needs to also ‘win’ the hardware lottery (Hooker, 2020), i.e., having readily supported ops, kernels or accelerator support to take advantage of its technical design.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Much of the recent revolution within machine learning, and in deep learning in particular, is in large part due to advancements in hardware (Hooker, 2020).”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, background): “Achieving all this together depended on the advent of a new neural architecture, the Transformer [184], which allowed model training to be scaled up in a way not possible with early recurrent neural networks [64].”
The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research (2020, methodology): “history, compute has played an important role in determining what counts as a breakthrough (Sutton, 2019) and which direction the research follows (Hooker, 2020).”
The AI gambit: leveraging artificial intelligence to combat climate change—opportunities, challenges, and recommendations (2021, methodology): “Modern AI research has focused on producing deeper and more accurate models at the detriment of energy efficiency (Sutton 2019; Perrault et al. 2019; Hooker 2020).”
Parameter Prediction for Unseen Deep Architectures (2021, background): “These nets are often more challenging to train for fundamental [49, 50] or computational [51] reasons, so predicting their parameters might ease their subsequent optimization.”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, background): “In fact, most sparse training methods slow down training in wall-clock time (Gale et al., 2019; Hooker, 2020).”
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models (2021, methodology): “…training methods require up to 5× more training epochs compared to dense models [Evci et al., 2020] 2An unstructured sparse model with 1% nonzero weights can be as slow as a dense model [Hooker, 2020] language modeling task, we speed up GPT-2 Medium training by 2.5× and achieve the same perplexity.”
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (2023, background): “…(Michel et al., 2019; Bansal et al., 2022), but maintaining different models for each task conflicts with the task independence goal of LLMs. Lastly, it is hard to achieve wall-clock time speed-up with unstructured sparsity due to its well-known difficulty with modern hardware (Hooker, 2021).”
Learning Mesh-Based Simulation with Graph Networks (2020, background): “While meshes are sometimes used for learned geometry processing [9] and generative models of shapes [13, 26], most work on predicting high-dimensional physical systems focuses on grids, owing to the popularity and hardware support for CNN architectures [17].”
Photorealistic Video Generation with Diffusion Models (2023, background): “Transformers [73] are highly scalable and parallelizable neural network architectures designed to win the hardware lottery [39].”
Everything is Connected: Graph Neural Networks (2023, background): “Accordingly, Transformers can be seen as GNNs that are currently winning the hardware lottery (Hooker, 2021)!”
The Benchmark Lottery (2021, methodology): “One of these factors is the software and hardware support for an idea, as captured in the concept of hardware lottery by Hooker [2020].”