Exploring Low-dimensional Intrinsic Task Subspace via Prompt Tuning

By Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Zhiyuan Liu, Juan-Zi Li, Lei Hou, Peng Li, Maosong Sun and 1 more. 2021

39 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Ning Ding
2. Zhiyuan Liu
3. Xipeng Qiu
4. Maosong Sun
5. Xu Han
6. Tianxiang Sun
7. Yankai Lin
8. Yujia Qin

Citations in the corpus

• ELLE: Efficient Lifelong Pre-training for Emerging Data (2022, background): “Soft prompts have been demonstrated as excellent task indicators (Qin et al., 2021b) and have non-trivial transferability among tasks (Su et al., 2021).”
• OpenPrompt: An Open-source Framework for Prompt-learning (2021, background): “These practical studies imply that we may use prompts to more effectively and efficiently dig the knowledge kept in PLMs, leading to a deeper understanding of the underlying principles of their mechanisms (Wei et al., 2021; Qin et al., 2021; Vu et al., 2021).”
• Parameter-efficient fine-tuning of large-scale pre-trained language models (2023, background): “Furthermore, intrinsic prompt-tuning 17 makes a stronger hypothesis that the adaptations to multiple tasks could be reparameterized into optimizations within the same low-dimensional intrinsic subspace.”
• ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation (2023, background): “The success of adapters in NLP has also led to widespread interest in applying adapter techniques to visual tasks.”
• On Transferability of Prompt Tuning for Natural Language Understanding (2021, methodology): “…and measuring the essential factors of prompts, which needs a deeper understanding on the mechanisms of PLMs. Previous works (Aghajanyan et al., 2021; Qin et al., 2021b) show that various NLP tasks can be reparameter-ized into similar low-dimensional subspaces conditioning on PLMs, which may help…”
• Black-Box Tuning for Language-Model-as-a-Service (2022, methodology): “As demonstrated in Qin et al. (2021), training the projection A with multi-task supervision can result in better and smaller subspace.”