Factorizing Knowledge in Neural Networks

By Xingyi Yang, Jingwen Ye, Xinchao Wang. 2022

136 citations recorded by the source corpus.

In this paper, we explore a novel and ambitious knowledge-transfer task, termed Knowledge Factorization~(KF). The core idea of KF lies in the modularization and assemblability of knowledge: given a pretrained network model as input, KF aims to decompose it into several factor networks, each of which handles only a dedicated task and maintains task-specific knowledge factorized from the source network. Such factor networks are task-wise disentangled and can be directly assembled, without any fine-tuning, to produce the more competent combined-task networks. In other words, the factor networks serve as Lego-brick-like building blocks, allowing us to construct customized networks in a plug-and-play manner. Specifically, each factor network comprises two modules, a common-knowledge module that is task-agnostic and shared by all factor networks, alongside with a task-specific module dedicated to the factor network itself. We introduce an information-theoretic objective, InfoMax-Bottleneck~(IMB), to carry out KF by optimizing the mutual information between the learned representations and input. Experiments across various benchmarks demonstrate that, the derived factor networks yield gratifying performances on not only the dedicated tasks but also disentanglement, while enjoying much better interpretability and modularity. Moreover, the learned common-knowledge representations give rise to impressive results on transfer learning. Our code is available at https://github.com/Adamdad/KnowledgeFactor.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Daquan Zhou
2. Xinchao Wang
3. Jiashi Feng
4. Dongze Lian
5. Gongfan Fang
6. Xinyin Ma
7. Dacheng Tao
8. Qibin Hou

Citations in the corpus

• LLM-Pruner: On the Structural Pruning of Large Language Models (2023, background): “With the growing size of models, there is an increasing demand for efficient LLM compression and compression is independent of the original training data.”
• Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning (2022, methodology): “Convolution has been used for a long time as the main module to extract the image features in computer vision tasks, and CNN-based architectures have been studied [49, 18, 59, 39, 60, 37, 63] with extension on graph-based data [62, 61, 36].”