Pretrained Transformers as Universal Computation Engines

By Kevin Lu, Aditya Grover, Pieter Abbeel, Igor Mordatch. 2021

238 citations recorded by the source corpus.

We investigate the capability of a transformer pretrained on natural language to generalize to other modalities with minimal finetuning -- in particular, without finetuning of the self-attention and feedforward layers of the residual blocks. We consider such a model, which we call a Frozen Pretrained Transformer (FPT), and study finetuning it on a variety of sequence classification tasks spanning numerical computation, vision, and protein fold prediction. In contrast to prior works which investigate finetuning on the same modality as the pretraining dataset, we show that pretraining on natural language can improve performance and compute efficiency on non-language downstream tasks. Additionally, we perform an analysis of the architecture, comparing the performance of a random initialized transformer to a random LSTM. Combining the two insights, we find language-pretrained transformers can obtain strong performance on a variety of non-language tasks.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Large Language Models

Citations in the corpus

Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021, positive): “Lu et al. (2021) show that Transformers can be used to some extent as universal computation engines, by pre-training on natu-ral language, and ﬁne-tuning a small fraction of the weights on non-language tasks, including simple algorithm induction tasks.”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “[13, 48]), but the limited scalability of Transformers restricts their usefulness as general-purpose architectures.”
Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning (2022, background): “Large language models (LLMs) are powerful few-shot learners (Bommasani et al., 2021; Brown et al., 2020; Lu et al., 2021).”
Pre-Trained Language Models for Interactive Decision-Making (2022, background): “, 2020) show that the GPT-3 model is capable of performing a limited set of arithmetic and string manipulation tasks; (Lu et al., 2021) show that pre-trained LMs require very little fine-tuning to match the performance of task-specific models on several image classification and numerical sequence pr”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “, 2021), and even as valuable priors when applied to diverse tasks from different modalities (Lu et al., 2021; Tsimpoukelli et al., 2021).”
Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers (2021, methodology): “Lu et al. (2021) pretrain a Transformer with a language modelling task.”
FLAVA: A Foundational Language And Vision Alignment Model (2021, background): “If the future of our field lies in generalized “foundation models” [10] or “universal” transformers [72] with many”
Multimodal Few-Shot Learning with Frozen Language Models (2021, background): “This finding extends the conclusion of [26] – that knowledge in transformer language models can transfer to non-linguistic tasks – to the specific case of knowledge about few-shot learning.”
PaLM-E: An Embodied Multimodal Language Model (2023, methodology): “, 2021) where vision encoder parameters are optimized via backpropagation through a frozen LLM (Lu et al., 2021).”
Disentangling Transfer in Continual Reinforcement Learning (2022, background): “Modern transformer-based models [45, 10] trained on large natural language corpora turned out to be very ﬂexible and can be adapted to diverse downstream tasks with surprising efﬁciency [34, 25].”
Can Wikipedia Help Offline Reinforcement Learning? (2022, background): “Previous work (Tsimpoukelli et al., 2021; Lu et al., 2021) has demonstrated how frozen language models have the capability to extend to the vision domain with respectable performance, which we aim to test with this experiment.”
Multimodal biomedical AI (2022, background): “As an example, some studies suggest that transformers pretrained on unlabeled language data might be able to generalize well to a broad range of other tasks 120 .”
Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, methodology): “In the pursuit of efficiency and practicality, most recent works have shifted towards parameter-efficient fine-tuning methods like Low-Rank Adaptation (LORA) [192] and Layer Normalization Tuning (LNT) [193].”
ProGen2: Exploring the Boundaries of Protein Language Models (2022, background): “In particular, the transformer (Vaswani et al., 2017) has proven to be a powerful language model and can serve as a universal computation engine (Lu et al., 2021) across data modalities.”
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes (2022, background): “There is a long line of work investigating the capabilities [69, 15, 77, 51, 76, 7, 78], limitations [25, 6], applications [37, 17, 49], and internal workings [19, 65, 71, 18, 48] of Transformer models.”
Understanding HTML with Large Language Models (2022, background): “It has only been in the last year that researchers have begun to utilize LLMs outside of NLP and integrate them as core capabilities in autonomy ((Lu et al., 2021; Ahn et al., 2022)).”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, methodology): “…techniques for reducing trainable parameters, including adapter-based ﬁne-tuning [25, 26, 27] that trains additional small residual blocks between layers, freezing-based ﬁne-tuning [71, 18, 72] that freezes most of the pretrained parameters, and distillation-based ﬁne-tuning [73].”
Grounding Language Models to Images for Multimodal Generation (2023, background): “Lu et al. (2022) show that language-pretrained transformers can transfer well to non-language tasks.”