Communication-Optimal Parallel Recursive Rectangular Matrix Multiplication

126 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Mesh-TensorFlow: Deep Learning for Supercomputers (2018, methodology): “Mesh-TensorFlow can express a wide range of uniform partitionings of the iteration space and therefore can adopt many best known mappings, e.g., 3D [3, 1] and 2.5D [8] algorithms for square matrices, CARMA [13] for rectangular matrices, 1.5D [18] algorithm for matrices with different sparsities, bes”
A massively parallel tensor contraction framework for coupled-cluster computations (2014, background): “Additional algorithmic considerations arise when we consider matrix multiplication of non-square matrices [44, 13].”
Communication optimal parallel multiplication of sparse random matrices (2013, methodology): “4) are adaptations of dense ones [13, 25], though an important distinction is that the sparse algorithms do not require extra local memory to minimize communication.”
Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability (2016, background): “Both implementations have an optimal performance for square matrices and suffer from the “degeneration” [5].”
Communication-Avoiding Optimization Methods for Distributed Massive-Scale Sparse Inverse Covariance Estimation (2017, background): “Speedups ranging from 2-100× have been reported in the literature [13, 24, 25, 38, 44, 6]; in fact, communication-avoiding algorithms decrease communication asymptotically in the amount of replication.”