Federated Learning Over Wireless Fading Channels

By Mohammad Mohammadi Amiri, Deniz Gündüz. 2019

626 citations recorded by the source corpus.

We study federated machine learning at the wireless network edge, where limited power wireless devices, each with its own dataset, build a joint model with the help of a remote parameter server (PS). We consider a bandwidth-limited fading multiple access channel (MAC) from the wireless devices to the PS, and propose various techniques to implement distributed stochastic gradient descent (DSGD) over this shared noisy wireless channel. We first propose a digital DSGD (D-DSGD) scheme, in which one device is selected opportunistically for transmission at each iteration based on the channel conditions; the scheduled device quantizes its gradient estimate to a finite number of bits imposed by the channel condition, and transmits these bits to the PS in a reliable manner. Next, motivated by the additive nature of the wireless MAC, we propose a novel analog communication scheme, referred to as the compressed analog DSGD (CA-DSGD), where the devices first sparsify their gradient estimates while accumulating error from previous iterations, and project the resultant sparse vector into a low-dimensional vector for bandwidth reduction. We also design a power allocation scheme to align the received gradient vectors at the PS in an efficient manner. Numerical results show that D-DSGD outperforms other digital approaches in the literature; however, in general the proposed CA-DSGD algorithm converges faster than the D-DSGD scheme, and reaches a higher level of accuracy. We have observed that the gap between the analog and digital schemes increases when the datasets of devices are not independent and identically distributed (i.i.d.). Furthermore, the performance of the CA-DSGD scheme is shown to be robust against imperfect channel state information (CSI) at the devices. Overall these results show clear advantages for the proposed analog over-the-air DSGD scheme, which suggests that learning and communication algorithms should be designed jointly to achieve the best end-to-end performance in machine learning applications at the wireless edge.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Harnessing Wireless Channels for Scalable and Privacy-Preserving Federated Learning (2020, methodology): “Applying compressive sensing techniques is promising, in which a sparsified update is encoded by multiplying a random matrix before transmission [26], and the received update is decoded using the approximate message passing (AMP) algorithm [27].”
Age-Based Scheduling Policy for Federated Learning in Mobile Edge Networks (2019, background): “This salient feature of on-device training brings along great advantages of eliminating the large communication overheads as well as preserving data privacy, and hence making FL particularly relevant for mobile applications [6–10].”
Client Selection and Bandwidth Allocation in Wireless Federated Learning Networks: A Long-Term Perspective (2020, background): “Recognizing the unique physical property of wireless transmission, [2], [20]–[22] propose analog model aggregation over the air, provided that a very stringent synchronization is available.”
Communication-Efficient Edge AI: Algorithms and Systems (2020, background): “• Fast aggregation via over-the-air computation [21], [84], [85], [86] • Aggregation frequency control with limited bandwidth and computation resources [87], [88], [89] • Data reshufﬂing via index coding and pliable index coding for improving training performance [90], [91], [92] • Straggler…”
Distributed Learning in Wireless Networks: Recent Progress and Future Challenges (2021, methodology): “Distributed learning requires efficient ways to perform distributed computation, both over-the-air [13]–[15] and off-the-air.”
Federated Learning in Mobile Edge Networks: A Comprehensive Survey (2019, methodology): “As an extension, the authors in [129] introduce error accumulation and gradient sparsification in addition to over-the-air computation.”
Federated Learning: A signal processing perspective (2021, background): “This may be exploited by applying random compressive linear precoding, which can be applied in fading channels without requiring the users to have channel state information, having the receiver recover the aggregated model via compressed sensing mechanisms [16].”
Semantic Communications for Future Internet: Fundamentals, Applications, and Challenges (2022, positive): “For example, in [180], gradient sparsification is adopted to compress the model at the transmitter by setting all but k elements with the highest magnitudes of entries to zero.”
One-Bit Over-the-Air Aggregation for Communication-Efficient Federated Edge Learning: Design and Convergence Analysis (2020, background): “Existing works on over-the-air aggregation, as well as its application in the context of FEEL [6], [7], [17]–[19] consider analog modulation assuming that the transmitter can modulate the carrier waveform as desired, freely choosing the I/Q coefficients as arbitrary real number.”
Federated Learning for Vehicular Internet of Things: Recent Advances and Open Issues (2020, methodology): “Amiri and Gunduz [62] Three techniques to achieve efﬁcient FL in a fading multiple access channel.”
Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications (2022, positive): “This ‘over-the-air computation (OAC)’ approach has received significant interest in recent years thanks to its bandwidth efficiency [263], [264], [268],”
Federated Machine Learning: Survey, Multi-Level Classification, Desirable Criteria and Future Directions in Communication and Networking Systems (2021, background): “The authors of [90] propose to compress the computed gradient to a finite number of bits and communicate these bits to the parameter server.”
Edge Learning for B5G Networks With Distributed Signal Processing: Semantic Communication, Edge Computing, and Wireless Sensing (2022, neutral): “On the other hand, the work in [154] considered the”
Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications (2021, methodology): “The convergence behavior demonstrates that the noisy iterates typically introduce non-negligible optimality gap in various FL algorithms, e.g., vanilla gradient method [174], quantized gradient method [175], sparsiﬁed gradient method [173], and operator splitting method [87].”
Machine Learning in the Air (2019, background): “This approach can also be extended to the scenario with fading [95], [96], in which case power control can be employed at the devices to align their transmissions at the same received power level.”
Holistic Network Virtualization and Pervasive Network Intelligence for 6G (2023, methodology): “Over-theair computation based approaches are investigated in [194], [216], [217].”