Parallel distributed computing using Python

By Lisandro Dalcin, Rodrigo Paz, P. Kler, Alejandro Cosimo. 2011

625 citations recorded by the source corpus.

Abstract This work presents two software components aimed to relieve the costs of accessing high-performance parallel computing resources within a Python programming environment: MPI for Python and PETSc for Python. MPI for Python is a general-purpose Python package that provides bindings for the Message Passing Interface (MPI) standard using any back-end MPI implementation. Its facilities allow parallel Python programs to easily exploit multiple processors using the message passing paradigm. PETSc for Python provides access to the Portable, Extensible Toolkit for Scientific Computation (PETSc) libraries. Its facilities allow sequential and parallel Python applications to exploit state of the art algorithms and data structures readily available in PETSc for the solution of large-scale problems in science and engineering. MPI for Python and PETSc for Python are fully integrated to PETSc-FEM, an MPI and PETSc based parallel, multiphysics, finite elements code developed at CIMEC laboratory. This software infrastructure supports research activities related to simulation of fluid flows with applications ranging from the design of microfluidic devices for biochemical analysis to modeling of large-scale stream/aquifer interactions.

Subfields: Artificial Intelligence, Biomedical Engineering, Hardware and Architecture

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Hardware and Architecture

Citations in the corpus

First Analysis of Local GD on Heterogeneous Data (2019, methodology): “We use a machine with 24 Intel(R) Xeon(R) Gold 6146 CPU @ 3.20GHz cores and we handle communication via the MPI for Python package [4].”
DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation (2019, methodology): “We implemented vanilla versions of the aforementioned Byzantine resilient methods, as well as versions of these methods pairing with D ETOX , in PyTorch [22] with MPI4py [23].”
DRACO: Byzantine-resilient Distributed Training via Redundant Gradients (2018, methodology): “, 2017b) with MPI4py (Dalcin et al., 2011) deployed on the m4.”
ATOMO: Communication-efficient Learning via Atomic Sparsification (2018, methodology): “Implementation and setup We compare spectral-ATOMO2 with different sparsity budgets to bbit QSGD across a distributed cluster with a parameter server (PS), implemented in mpi4py [49] and PyTorch [50] and deployed on multiple types of instances in Amazon EC2 (e.”
Tighter Theory for Local SGD on Identical and Heterogeneous Data (2019, methodology): “The code was written in Python using MPI (Dalcin et al., 2011) and run on Intel(R) Xeon(R) Gold 6146 CPU @3.”
Stochastic Distributed Learning with Gradient Quantization and Variance Reduction (2019, methodology): “We implement all methods in Python using MPI4PY (Dalcin et al., 2011) for collective communication operations.”
Gradient Coding: Avoiding Stragglers in Distributed Learning (2017, methodology): “Experimental setup: We implemented all methods in python using MPI4py (Dalcin et al., 2011), an open source MPI implementation.”
Election Coding for Distributed Learning: Protecting SignSGD against Byzantine Attacks (2019, methodology): “We implement the suggested coded distributed learning algorithms in PyTorch, and deploy them on Amazon EC2 using Python with MPI4py package.”
The FEniCS Project Version 1.5 (2015, methodology): “6; provides Python bindings to SLEPc [Dalcin et al., 2011] • UMFPACK; provides sequential sparse LU decomposition using multifrontal method [Davis, 2004]”
A spike sorting toolbox for up to thousands of electrodes validated with ground truth recordings in vitro and in vivo (2018, methodology): “SpyKING CIRCUS is a pure Python package, based on the python wrapper for the Message Passing Interface (MPI) library (Dalcin et al., 2011) to allow parallelization over distributed computers, and is available with its full documentation at http://spyking-circus.”
Near-Optimal Straggler Mitigation for Distributed Gradient Methods (2017, methodology): “We use Python as our programming language and MPI4py [15] for message passing across EC2 instances.”
99% of Parallel Optimization is Inevitably a Waste of Time (2019, background): “Hopefully, the parameter server can copy the current vector and send it using non-blocking communication, such as isend() in MPI4PY (Dalcin et al., 2011).”
99% of Distributed Optimization is a Waste of Time: The Issue and How to Fix it (2019, background): “[7] Lisandro D Dalcin, Rodrigo R Paz, Pablo A Kler, and Alejandro Cosimo.”
Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics (2019, methodology): “The elements of the parallel architecture are trained independently, while the MPI (Dalcín, Paz, Kler, & Cosimo, 2011; Dalcín, Paz, Storti, & D’Elía, 2008; Walker & Dongarra, 1996) communication protocol is utilized to communicate the elements of the interaction for long-term forecasting.”
Gradient Coding (2016, methodology): “We implemented the distributed schemes in python using MPI4py[3], an open source MPI implementation.”
Better Communication Complexity for Local SGD (2019, methodology): “The code was written in Python using MPI [6].”