Adaptive Gradient-Based Meta-Learning Methods

By Mikhail Khodak, Maria-Florina Balcan, Ameet Talwalkar. 2019

397 citations recorded by the source corpus.

We build a theoretical framework for designing and understanding practical meta-learning methods that integrates sophisticated formalizations of task-similarity with the extensive literature on online convex optimization and sequential prediction algorithms. Our approach enables the task-similarity to be learned adaptively, provides sharper transfer-risk bounds in the setting of statistical learning-to-learn, and leads to straightforward derivations of average-case regret bounds for efficient algorithms in settings where the task-environment changes dynamically or the tasks share a certain geometric structure. We use our theory to modify several popular meta-learning algorithms and improve their meta-test-time performance on standard problems in few-shot learning and federated learning.

Subfields: Artificial Intelligence, Computational Mechanics, Optimization & Training, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Optimization & Training

In Meta & Continual Learning

Citations in the corpus

Advances and Open Problems in Federated Learning (2019, background): “[234] and Jiang et al.”
Federated Optimization in Heterogeneous Networks (2018, background): “Finally, we note that there is an interesting connection between meta-learning methods and federated optimization methods (Khodak et al., 2019), and similar proximal terms have recently been investigated in the context of meta-learning for improved performance on few-shot learning tasks (Goldblum…”
FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “In terms of network topology, a variety of network typologies such as vertical FL [34, 35, 36, 37, 38, 39, 40], split learning [41, 42], decentralized FL [43, 44, 45, 46], hierarchical FL [47, 48, 49, 50, 51, 52], and meta FL [53, 54, 55] have been proposed.”
On the Outsized Importance of Learning Rates in Local Update Methods (2020, background): “Connections between these two areas were noted by Jiang et al. (2019) and have since been explored in many other works (Khodak et al., 2019; Fallah et al., 2020).”
Personalized Federated Learning with Moreau Envelopes (2020, methodology): “[35] based its framework, ARUBA, on online convex optimization and meta-learning, which can be integrated into FL to improve personalization.”
Three Approaches for Personalization with Applications to Federated Learning (2020, background): “, 2017), meta-learning (Jiang et al., 2019; Khodak et al., 2019), use of local parameters (Arivazhagan et al.”
Personalized Federated Learning: A Meta-Learning Approach (2020, methodology): “Khodak et al. (2019) proposed ARUBA, a meta-learning al-gorithm inspired by online convex optimization, and showed how applying it to FedAvg method improves its performance empirically.”
Improving Federated Learning Personalization via Model Agnostic Meta Learning (2019, background): “…as meta-training in the MAML language, and the FL personalization via gradient descent as meta-testing, we show in Section 2 that FedAvg (McMahan et al., 2017) and Reptile (Nichol et al., 2018), two popular FL and MAML algorithms, are very similar to each other; see also Khodak et al. (2019).”
Differentially Private Meta-Learning (2019, methodology): “Our learning guarantees scale with task-similarity, as measured by the closeness of the task-specific optimal parameters (Denevi et al., 2019; Khodak et al., 2019b).”
LEAF: A Benchmark for Federated Settings (2018, background): “These particular pipelines shed light on how different modeling approaches may be more or less appropriate for different federated datasets [10, 12].”
Federated Learning: Challenges, Methods, and Future Directions (2019, background): “Modeling heterogeneous data There exists a large body of literature in machine learning that models statistical heterogeneity via methods such as metalearning and multitask learning; these ideas have been recently extended to the federated setting [12], [14], [21].”
Adaptive Personalized Federated Learning (2020, methodology): “A similar quantity is also employed in the related multiple source PAC learning or domain adaption studies Kifer et al. (2004); Mansour et al.”
FedMD: Heterogenous Federated Learning via Model Distillation (2019, methodology): “Approaches based on frameworks such as Bayesian [11], meta-learning [12] and transfer learning [14] also achieve good performance on non-i.i.d. data while allowing a certain amount of model customization.”
CoMPS: Continual Meta Policy Search (2021, background): “Recent work has made progress toward using supervised meta-learning in an online setting (Ren et al., 2018; Finn et al., 2019; Khodak et al., 2019; Jerfel et al., 2019; Grant et al., 2019; Zhuang et al., 2019; Antoniou et al., 2020; Wallingford et al., 2020; Yu et al., 2020a; Yao et al., 2020).”
Few-Shot Learning via Learning the Representation, Provably (2020, background): “Another recent line of theoretical work analyzed gradient-based meta-learning methods (Denevi et al., 2019, Finn et al., 2019, Khodak et al., 2019) and showed guarantees for convex losses by using tools from online convex optimization.”
Federated Learning of a Mixture of Global and Local Models (2020, methodology): “(Khodak et al., 2019) propose a generic online algorithm for gradient-based parameter-transfer meta-learning and demonstrate improved practical performance over FedAvg (McMahan et al., 2017).”
Federated Mutual Learning (2020, background): “ a bug in training processing[2]. Smith et al. [12] introduce a MOCHA framework for multi-task federated learning, dealing with high communication cost, stragglers, and fault tolerance. Khodak et al. [13] presents an Average RegretUpper-Bound Analysis (ARUBA) theoretical framework for analyzing grad”
Exploiting Shared Representations for Personalized Federated Learning (2021, methodology): “…using, for example, local ﬁne-tuning [Wang et al., 2019, Yu et al., 2020], meta-learning [Chen et al., 2018, Fallah et al., 2020, Jiang et al., 2019, Khodak et al., 2019], additive mixtures of local and global models [Deng et al., 2020, Hanzely and Richt´arik, 2020, Mansour et al., 2020], and…”
Personalized Federated Learning: An Attentive Collaboration Approach (2020, background): “Similarly, some meta learning methods [9, 15, 16, 23, 24, 25] can be extended to build personalized models by adapting a trained global model on the local data of a client with a few ﬁne-tuning steps [8].”