Communication-Avoiding Optimization Methods for Distributed Massive-Scale Sparse Inverse Covariance Estimation
By Penporn Koanantakool, Alnur Ali, Ariful Azad, Aydin Buluç, Dmitriy Morozov, Leonid Oliker, K. Yelick, Sang-Yun Oh. 2017
12 citations recorded by the source corpus.
Author(s): Koanantakool, P; Ali, A; Azad, A; Buluc, A; Morozov, D; Oliker, L; Yelick, K; Oh, SY | Abstract: Copyright 2018 by the author(s). Across a variety of scientific disciplines, sparse inverse covariance estimation is a popular tool for capturing the underlying dependency relationships in multivariate data. Unfortunately, most estimators are not scalable enough to handle the sizes of modern high-dimensional data sets (often on the order of terabytes), and assume Gaussian samples. To address these deficiencies, we introduce HP-CONCORD, a highly scalable optimization method for estimating a sparse inverse covariance matrix based on a regularized pseudolikelihood framework, without assuming Gaussianity. Our parallel proximal gradient method uses a novel communication-avoiding linear algebra algorithm and runs across a multi-node cluster with up to 1k nodes (24k cores), achieving parallel scalability on problems with up to ≈819 billion parameters (1.28 million dimensions); even on a single node, HP-CONCORD demonstrates scalability, outperforming a state-of-the-art method. We also use HP-CONCORD to estimate the underlying dependency structure of the brain from fMRI data, and use the result to identify functional regions automatically. The results show good agreement with a clustering from the neuroscience literature.
Subfields: Cognitive Neuroscience, Computational Mechanics, Statistical and Nonlinear Physics, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Cognitive Neuroscience
In Computational Mechanics
In Statistical and Nonlinear Physics
- HyoukJoong Lee
- Ashish Vaswani
- Blake A. Hechtman
- Noam Shazeer
- Dustin Tran
- Quoc V. Le
- Orhan Firat
- Neil Houlsby
In Optimization & Training
- HyoukJoong Lee
- Ashish Vaswani
- Blake A. Hechtman
- Noam Shazeer
- Peter G. Hawkins
- Mingsheng Hong
- Penporn Koanantakool
- Ryan Sepassi
Citations in the corpus
- Mesh-TensorFlow: Deep Learning for Supercomputers (2018, background): “Furthermore, in most existing work, when multiple multiplications are composed together, the user has to specify the data layout for each matrix separately [22].”