Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air

By Mohammad Mohammadi Amiri, Deniz Gündüz. International Symposium on Information Theory, 2019

591 citations recorded by the source corpus.

We study collaborative machine learning at the wireless edge, where power and bandwidth-limited devices (workers), with limited local datasets, implement distributed stochastic gradient descent (DSGD) over-the-air with the help of a remote parameter server (PS). We consider a wireless multiple access channel (MAC) from the workers to the PS for communicating the local gradient estimates. We first introduce a digital DSGD (D-DSGD) scheme, assuming that the workers operate on the boundary of the MAC capacity region at each iteration of the DSGD algorithm, and digitize their estimates within the bit budget allowed by the employed power allocation. We then introduce an analog scheme, called A-DSGD, motivated by the additive nature of the wireless MAC, where the workers send their gradient estimates over the MAC through the available channel bandwidth without employing any digital code. Numerical results show that A-DSGD converges much faster than D-DSGD. The improvement is particularly compelling at low power and low bandwidth regimes. We also observe that the performance of A-DSGD improves with the number of workers, while D-DSGD deteriorates, limiting the ability of the latter in harnessing the computation power of many edge devices.

Subfields: Artificial Intelligence, Computer Networks and Communications

View Paper: https://www.semanticscholar.org/paper/eeaedfddff1e377e4eabaeb0e45cab4e9e5d6f65

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

Citations in the corpus

Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications (2021, methodology): “To tackle the channel fading perturbation, a channel inversion method was proposed in [45], [173], [44] by multiplying the inverse of channel gain for the transmit signal, which may however not satisfy the power constraint at edge devices.”
Machine Learning in the Air (2019, background): “7, where the transmission of local gradient estimates from the workers to the parameter server can be formulated as a wireless computation problem [77].”
Hierarchical Federated Learning ACROSS Heterogeneous Cellular Networks (2019, background): “To this end, there are some very recent works that focus on the distributed machine learning framework with a particular focus on wireless communications [1]–[3], [7]–[13].”
On Analog Gradient Descent Learning Over Multiple Access Fading Channels (2019, background): “In [8]–[10], the authors developed the compressed analog distributed stochastic gradient descent (CA-DSGD) algorithm, in which each node transmits a sparse parameter gradient vector over MAC.”
Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications (2022, unknown): “This ‘over-the-air computation (OAC)’ approach has received significant interest in recent years thanks to its bandwidth efficiency [263], [264], [268],”
Federated Learning With Quantized Global Model Updates (2020, background): “Several studies address this issue by considering communications with rate limitations, and propose diﬀerent compression and quantization techniques [2–9], as well as performing local updates to reduce the frequency of communications from the devices to the PS [10, 11].”
Federated Learning With Cooperating Devices: A Consensus Approach for Massive IoT Networks (2019, methodology): “Application of consensus based federated optimization to deeper networks might require a more efficient use of the limited bandwidth, including quantization, compression or ad-hoc channel encoding [47].”
Federated Learning: A signal processing perspective (2021, background): “, local SGD with E = 1) precoded to meet a given power constraint was shown to result in low convergence delay when learning is carried out over identical unfaded channels with homogeneous datasets [30].”
Federated Learning in Vehicular Networks (2020, background): “exploited to reduce the amount of transmitted data [15].”
Federated Learning Over Wireless Fading Channels (2019, methodology): “Finally, we propose our scheme, built upon our previous work [25], referred to as the compressed analog DSGD (CA-DSGD) scheme.”
One-Bit Over-the-Air Aggregation for Communication-Efficient Federated Edge Learning: Design and Convergence Analysis (2020, methodology): “Existing works on over-the-air aggregation, as well as its application in the context of FEEL [6], [7], [17]–[19] consider analog modulation assuming that the transmitter can modulate the carrier waveform as desired, freely choosing the I/Q coefficients as arbitrary real number.”
Communication-efficient federated learning (2021, unknown): “However, the majority of these works focus on optimization of FL in a single aspect such as device selection and scheduling (11–13), FL model parameter update and transmission (14–18), or network resource management (20–22).”
Over-the-Air Machine Learning at the Wireless Edge (2019, methodology): “Here, we extend our previous work in [12] that focused on DSGD over a Gaussian MAC to a fading MAC.”
Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Data (2019, methodology): “Accordingly, each device k at the i-th global iteration computes the vector v i = sparseq ( ∆w i + ∆ k i ) , where the accumulated quantization error is updated as [10], [11] ∆ i = ∆w k i + ∆ k i −Qb ( v i ) .”
Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing (2019, methodology): “[57] M. M. Amiri and D. Gunduz, “Machine learning at the wireless edge: Distributed stochastic gradient descent over-the-air,” 2019, arXiv:1901.00844.”