LEAF: A Benchmark for Federated Settings
By Sebastian Caldas, Peter Wu, Tian Li, Jakub Konecný, H. Brendan McMahan, Virginia Smith, Ameet Talwalkar. 2018
1,782 citations recorded by the source corpus.
Modern federated networks, such as those comprised of wearable devices, mobile phones, or autonomous vehicles, generate massive amounts of data each day. This wealth of data can help to learn models that can improve the user experience on each device. However, the scale and heterogeneity of federated data presents new challenges in research areas such as federated learning, meta-learning, and multi-task learning. As the machine learning community begins to tackle these challenges, we are at a critical time to ensure that developments made in these areas are grounded with realistic benchmarks. To this end, we propose LEAF, a modular benchmarking framework for learning in federated settings. LEAF includes a suite of open-source federated datasets, a rigorous evaluation framework, and a set of reference implementations, all geared towards capturing the obstacles and intricacies of practical federated environments.
Subfields: Artificial Intelligence, Evaluation & Benchmarks, Federated & Distributed Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Qiang Yang
- Dacheng Tao
- Percy Liang
- Yoshua Bengio
- Sergey Levine
- Martin Jaggi
- Ruslan Salakhutdinov
- Quoc V. Le
In Evaluation & Benchmarks
- Ameet Talwalkar
- Virginia Smith
- Peter Kairouz
- Zachary B. Charles
- Tian Li
- Jakub Konecný
- Chelsea Finn
- Percy Liang
In Federated & Distributed Learning
Citations in the corpus
- Federated Optimization in Heterogeneous Networks (2018, methodology): “Finally, we note that since FedProx makes only lightweight modifications to FedAvg, this allows us to reason about the behavior of the widely-used FedAvg method, and enables easy integration of FedProx into existing packages/systems, such as TensorFlow Federated and LEAF (TFF; Caldas et al., 2018).”
- Ensemble Distillation for Robust Model Fusion in Federated Learning (2020, background): “Federated Learning (FL) has emerged as an important machine learning paradigm in which a federation of clients participate in collaborative training of a centralized model (Shokri & Shmatikov, 2015; McMahan et al., 2016; Smith et al., 2017; Caldas et al., 2018; Bonawitz et al., 2019; Li et al., 2019”
- The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation (2021, methodology): “We evaluate on two realistic FL tasks: Federated EMNIST (Cohen et al., 2017; Caldas et al., 2018) and Stack 6The kinks on the low bit-width curves are due to the TensorFlow implementation of the discrete Gaussian sampler taking integer noise scales; to preserve privacy, noise scales are rounded up a”
- Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning (2020, methodology): “We run experiments on simulated and real (EMNIST62 and CIFAR100) datasets to confirm our theory.”
- Flower: A Friendly Federated Learning Research Framework (2020, methodology): “We therefore apply the concept of partial work (Li et al., 2018) in which a client submits their locally updated θ k upon reaching ∆ along with c k , the number of examples actually involved in computing θ k , even if c k E m k B B .”
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, methodology): “The federated version of EMNIST [72] partitions the digits by their author.”
- On the Outsized Importance of Learning Rates in Local Update Methods (2020, methodology): “We also see that while suboptimality gaps exist for FEMNIST and Shakespeare, they are much smaller than for CIFAR-100.”
- Adaptive Federated Optimization (2020, methodology): “The federated version of EMNIST (Caldas et al., 2018) partitions the digits by their author.”
- Three Approaches for Personalization with Applications to Federated Learning (2020, methodology): “We evaluate the proposed algorithms on the federated EMNIST-62 dataset ( Caldas et al., 2018) provided by TensorFlow Federated (TFF).”
- Robust Aggregation for Federated Learning (2019, methodology): “The simulations were run using TensorFlow and the data was preprocessed using LEAF [19].”
- Can You Really Backdoor Federated Learning? (2019, methodology): “In the above backdoor attack framework, we conduct experiments on the EMNIST dataset (11; 9).”
- Generative Models for Effective ML on Private, Decentralized Datasets (2019, methodology): “We use the Federated EMNIST dataset (Caldas et al., 2018) to represent the app’s users’ processed data in our GAN experiments in Section 6.”
- Secure Federated Submodel Learning (2019, background): “[41] released a benchmark for federated learning, called LEAF.”
- Real-World Image Datasets for Federated Learning (2019, background): “…federated learning system design in 2019 [ Bonawitz et al. , 2019 ] , highlighting some of the design challenges for large-scale problems. et al. , 2018] is a benchmark framework that contains preprocessed datasets, each with a “natural” partitioning that aims to reflect the type of…”
- The Non-IID Data Quagmire of Decentralized Machine Learning (2019, background): “For example, LEAF (Caldas et al., 2018) provides datasets that are partitioned in various ways.”
- Improving Federated Learning Personalization via Model Agnostic Meta Learning (2019, methodology): “We suggest these objectives become the subject of MAML works, in addition to the performance after adaptation, and to consider the datasets with a natural user/client structure being established for Federated Learning (Caldas et al., 2018b) as the source of experiments for supervised learning.”
- Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification (2019, methodology): “For example, Caldas et al. [2018] use Extended MNIST [Cohen et al., 2017] with partitions over writers of the digits, rather than simply partitioning over digit class.”
- Differentially Private Meta-Learning (2019, methodology): “Datasets: We train a next word predictor for two federated datasets: (1) The Shakespeare dataset as preprocessed by (Caldas et al., 2018), and (2) a dataset constructed from 3, 000 Wikipedia articles drawn from the Wiki-3029 dataset (Arora et al.”
- Adaptive Gradient-Based Meta-Learning Methods (2019, methodology): “Figure 3: Next-character prediction performance for recurrent networks trained on the Shakespeare dataset [12] using FedAvg [41] and its modifications by Algorithm 2.”