MPI: A message - passing interface standard

By Message Passing Interface Forum. 1994

Subfields: ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In ML Systems & Hardware

1. K. Hebeler
2. T. E. Riley
3. R. Ludlam
4. S. K. Greif
5. S. Nissanke
6. Jean-Yves L'Excellent
7. Rodrigo Paz
8. Xin Yuan

Citations in the corpus

• Pathways: Asynchronous Distributed Dataflow for ML (2022, background): “Dedicated GPU and TPU interconnects are typically exposed to applications via 30 year old MPI primitives (e.g., AllReduce) that must be gang-scheduled so that every program enters the same primitive at the same time.”
• GSPMD: General and Scalable Parallelization for ML Computation Graphs (2021, methodology): “We use XLA’s operators that provide MPI-style collective communications [23].”
• Efficiently Scaling Transformer Inference (2022, positive): “We use several communication collectives originating from MPI (Clarke et al., 1994).”
• Communication-Efficient Distributed Optimization of Self-Concordant Empirical Loss (2015, neutral): “O C ] 1 Ja”
• JuMP: A Modeling Language for Mathematical Optimization (2015, methodology): “This ability to generate the model in parallel (in a distributed-memory MPI-based [61] fashion) distinguishes StructJuMP from existing tools such as PySP [80].”
• Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models (2022, methodology): “MPI-style collective operations [21] are used to represent communication that occurs in intra-layer model parallelism with single program, multiple data (SPMD) paradigm.”
• A Class of Parallel Tiled Linear Algebra Algorithms for Multicore Architectures (2007, methodology): “…scalability limits in the context of multicore computing, Section 3 describes fine granularity, tiled algorithms for the Cholesky, LU and QR factorizations and presents a programming model for their asynchronous and dynamic execution; performance results for this algorithm are shown in Section 5.”
• Multifrontal parallel distributed symmetric and unsymmetric solvers (2000, methodology): “Our module provides an equivalent of MPI BSEND (Dongarra, Hempel, Hey and Walker 1995) with the advantage that messages are directly packed in the bu(cid:11)er and problems occurring when the bu(cid:11)er is full are overcome.”