Distributed Weight Consolidation: A Brain Segmentation Case Study

By Patrick McClure, Charles Y. Zheng, Jakub R. Kaczmarzyk, John Rogers-Lee, Satrajit S. Ghosh, Dylan M. Nielson, Peter A. Bandettini, Francisco Pereira. 2018

28 citations recorded by the source corpus.

Collecting the large datasets needed to train deep neural networks can be very difficult, particularly for the many applications for which sharing and pooling data is complicated by practical, ethical, or legal concerns. However, it may be the case that derivative datasets or predictive models developed within individual sites can be shared and combined with fewer restrictions. Training on distributed data and combining the resulting networks is often viewed as continual learning, but these methods require networks to be trained sequentially. In this paper, we introduce distributed weight consolidation (DWC), a continual learning method to consolidate the weights of separate neural networks, each trained on an independent dataset. We evaluated DWC with a brain segmentation case study, where we consolidated dilated convolutional neural networks trained on independent structural magnetic resonance imaging (sMRI) datasets from different sites. We found that DWC led to increased performance on test sets from the different sites, while maintaining generalization performance for a very large and completely independent multi-site dataset, compared to an ensemble baseline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Subspace Regularizers for Few-Shot Class Incremental Learning (2021, background): “…to recognize new concepts (e.g. new skills in personal assistants or new road signs in self-driving vehicles), while training data is sometimes unavailable for reuse (e.g. due to privacy regulations, Lesort et al. 2019; McClure et al. 2018; or storage and retraining costs, Bender et al. 2021).”
Class-Incremental Learning: Survey and Performance Evaluation on Image Classification (2020, background): “Privacy considerations are also common in health applications where legislation prevents the long-term storage of data from patients, and thus incremental learning is key [15].”
Automated assessment of psychiatric disorders using speech: A systematic review (2019, background): “33 Distributed training 125 is a more novel approach where the actual model is trained on the participant's device, and then the learned weights or configuration — but not the data — is returned to the researcher, a training style which can now be done with tools such as TensorFlow.js. 126”
Model Aggregation via Good-Enough Model Spaces (2018, background): “This scenario can face many of the challenges described above, as hospitals operate under strict privacy practices, and may face legal, administrative, or ethical constraints prohibiting a shared connected network for training [3, 16, 23, 28].”