Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning
By Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, Fei Huang. 2021
228 citations recorded by the source corpus.
Recent pretrained language models extend from millions to billions of parameters. Thus the need to fine-tune an extremely large pretrained model with a limited training corpus arises in various downstream tasks. In this paper, we propose a straightforward yet effective fine-tuning technique, Child-Tuning, which updates a subset of parameters (called child network) of large pretrained models via strategically masking out the gradients of the non-child network during the backward process. Experiments on various downstream tasks in GLUE benchmark show that Child-Tuning consistently outperforms the vanilla fine-tuning by 1.5 8.6 average score among four different pretrained models, and surpasses the prior fine-tuning techniques by 0.6 1.3 points. Furthermore, empirical results on domain transfer and task transfer show that Child-Tuning can obtain better generalization performance by large margins.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, neutral): “Invasive meth-ods, which are built on a strong assumption that the inner structure (e.g., self-attention and feed-forward layers) of the PLM can be modified, includes Prefix-Tuning (Li and Liang, 2021), Bit-fit (Ben Zaken et al., 2021), Child-Tuning (Xu et al., 2021), P-Tuning v2 (Liu et al., 2021b), L”
- How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2023, background): “Many efforts have been made to compensate for the phenomenon (Liang et al., 2021; Xu et al., 2021; Yuan et al., 2023a).”
- On the Effectiveness of Parameter-Efficient Fine-Tuning (2022, methodology): “2022) and even achieve better overall scores than the fully fine-tuned models (Lee, Cho, and Kang 2019; Houlsby et al. 2019; Zaken, Ravfogel, and Goldberg 2021; Sung, Nair, and Raffel 2021; Liu et al. 2021; Xu et al. 2021; Guo, Rush, and Kim 2021; He et al. 2021a; Ding et al. 2022) on some tasks.”
- ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge (2023, methodology): “We used conversations from HealthCareMagic-100k to fine-tune the LLaMA model [15] in line with Stanford Alpaca [5] training methodology.”
- PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, unknown): “PEFT encompasses strategies like partial fine-tuning [15, 16, 17, 18, 19, 20, 21, 22], soft prompt fine-tuning [23, 24, 25, 26, 27, 28, 29], non-linear adapter fine-tuning [30, 31, 32, 33, 34, 35, 36, 37, 38, 39], and low rank adapter based fine-tuning [40, 41, 11, 42].”
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “U-Diff pruning [59], U-BitFit [99], PaFi [111], FishMask [169], Fish-Dip [34], LT-SFT [3], SAM [48], Child-tuning [194] Structural Masking S-Diff pruning [59], S-BitFit [99], FAR [174], Bitfit [207], Xattn Tuning [54], SPT [66] ReparameterizedFine-tuning”
- Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, methodology): “In contrast, PEFT methods aim to alleviate these requirements by selectively updating or modifying specific parts of the PLMs while still achieving performance comparable to full fine-tuning [34], [39].”
- Leveraging Large Language Models for Mental Health Prediction via Online Text Data (2023, background): “3.2, the goal of this strategy is closer to the traditional few-shot transfer learning, where we further train the model with a small amount of domain-specific data ( e.g. , [40, 50, 89]).”