Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis

By A. Reuther, J. Kepner, C. Byun, S. Samsi, W. Arcand, David Bestor, Bill Bergeron, Vijay Gadepally, Michael Houle, M. Hubbell and 8 more. 2018

338 citations recorded by the source corpus.

Interactive massively parallel computations are critical for machine learning and data analysis. These computations are a staple of the MIT Lincoln Laboratory Supercomputing Center (LLSC) and has required the LLSC to develop unique interactive supercomputing capabilities. Scaling interactive machine learning frameworks, such as TensorFlow, and data analysis environments, such as MATLAB/Octave, to tens of thousands of cores presents many technical challenges – in particular, rapidly dispatching many tasks through a scheduler, such as Slurm, and starting many instances of applications with thousands of dependencies. Careful tuning of launches and prepositioning of applications overcome these challenges and allow the launching of thousands of tasks in seconds on a 40,000-core supercomputer. Specifically, this work demonstrates launching 32,000 TensorFlow processes in 4 seconds and launching 262,000 Octave processes in 40 seconds. These capabilities allow researchers to rapidly explore novel machine learning architecture and data analysis algorithms.

Subfields: Computer Networks and Communications, Hardware and Architecture, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Hardware and Architecture

In Information Systems

Citations in the corpus

Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, methodology): “Our work would also not have been possible without the excellent TransformerLens library [85] and the computing resources provided by the MIT supercloud [86].”
OSQP: an operator splitting solver for quadratic programs (2017, methodology): “All the experiments were carried out on the MIT SuperCloud facility in collaboration with the Lincoln Laboratory [83] with 16 Intel Xeon E5-2650 cores.”
A large-scale benchmark for few-shot program induction and synthesis (2021, methodology): “To efficiently evaluate these programs we leveraged the MIT supercloud (Reuther et al., 2018), parallelizing program evaluations over 4800 CPU cores.”
EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations (2023, neutral): “We acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center [69] for providing high performance computing and consultation resources that have contributed to the research results reported within this paper.”
From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference (2023, methodology): “We conducted our experiments on the MIT Supercloud high-performance computing (HPC) system [18].”
Chemprop: A Machine Learning Package for Chemical Property Prediction (2023, methodology): “47 For both systems, we restricted the maximum numbers of CPU cores accessible to Chemprop to 8.”
SketchOGD: Memory-Efficient Continual Learning (2023, neutral): “The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center [33] for providing computing resources that have contributed to the research results reported within this paper.”
Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs (2022, methodology): “We acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center [58] for providing high performance computing and consultation resources that have contributed to the research results reported within this paper.”
Learning to Recombine and Resample Data for Compositional Generalization (2020, background): “The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center (Reuther et al., 2018) for providing HPC resources that have contributed to the research results reported within this paper.”