LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks
By Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Shashank Rajput, Michael Gira, Jy-yong Sohn, Dimitris Papailiopoulos, Kangwook Lee. 2022
200 citations recorded by the source corpus.
Fine-tuning pretrained language models (LMs) without making any architectural changes has become a norm for learning various language downstream tasks. However, for non-language downstream tasks, a common practice is to employ task-specific designs for input, output layers, and loss functions. For instance, it is possible to fine-tune an LM into an MNIST classifier by replacing the word embedding layer with an image patch embedding layer, the word token output layer with a 10-way output layer, and the word prediction loss with a 10-way classification loss, respectively. A natural question arises: Can LM fine-tuning solve non-language downstream tasks without changing the model architecture or loss function? To answer this, we propose Language-Interfaced Fine-Tuning (LIFT) and study its efficacy and limitations by conducting an extensive empirical study on a suite of non-language classification and regression tasks. LIFT does not make any changes to the model architecture or loss function, and it solely relies on the natural language interface, enabling"no-code machine learning with LMs."We find that LIFT performs comparably well across a wide range of low-dimensional classification and regression tasks, matching the performances of the best baselines in many cases, especially for the classification tasks. We also report experimental results on the fundamental properties of LIFT, including inductive bias, robustness, and sample complexity. We also analyze the effect of pretraining on LIFT and a few properties/techniques specific to LIFT, e.g., context-aware learning via appropriate prompting, calibrated predictions, data generation, and two-stage fine-tuning. Our code is available at https://github.com/UW-Madison-Lee-Lab/LanguageInterfacedFineTuning.
Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Vadim Borisov
- Gjergji Kasneci
- Tobias Leemann
- Kathrin Sessler
- Martin Pawelczyk
- Gerhard Weikum
- Fabian M. Suchanek
- Thomas Neumann
In Large Language Models
Citations in the corpus
- TabLLM: Few-shot Classification of Tabular Data with Large Language Models (2022, unknown): “These findings are partly consistent with Dinh et al. (2022) who used GPT-3 and tested serializations with removed or permuted column names.”
- Leveraging large language models for predictive chemistry (2024, unknown): “Recently, it has been shown that these models can also solve relatively simple tabular regression and classification tasks 13 .”
- Language Models are Realistic Tabular Data Generators (2022, background): “This is even true in the one-shot setting for inference tasks (Dinh et al., 2022).”