Are Pretrained Convolutions Better than Pretrained Transformers?
By Yi Tay, Mostafa Dehghani, Jai Gupta, Dara Bahri, Vamsi Krishna Aribandi, Zhen Qin, Donald Metzler. 2021
54 citations recorded by the source corpus.
In the era of pre-trained language models, Transformers are the de facto choice of model architectures. While recent research has shown promise in entirely convolutional, or CNN, architectures, they have not been explored using the pre-train-fine-tune paradigm. In the context of language models, are convolutional models competitive to Transformers when pre-trained? This paper investigates this research question and presents several interesting findings. Across an extensive set of experiments on 8 datasets/tasks, we find that CNN-based pre-trained models are competitive and outperform their Transformer counterpart in certain scenarios, albeit with caveats. Overall, the findings outlined in this paper suggest that conflating pre-training and architectural advances is misguided and that both advances should be considered independently. We believe our research paves the way for a healthy amount of optimism in alternative architectures.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yi Tay
- Mostafa Dehghani
- Quoc V. Le
- Donald Metzler
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Andrew Zisserman
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Ming-Hsuan Yang
- Cordelia Schmid
Citations in the corpus
- Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, methodology): “We report all three key4 efficiency metrics (number of params, 3Our concurrent work Charformer (Tay et al., 2021b) makes use of a DeepNarrow inspired strategy which is referred to as Tall in the paper.”
- UL2: Unifying Language Learning Paradigms (2022, methodology): “This has become standard and defacto setup (Raffel et al., 2019; He et al., 2022; Tay et al., 2020; 2021b).”
- Do Vision Transformers See Like Convolutional Neural Networks? (2021, background): “Comparison between Transformers and CNNs are also recently studied in the text domain [41].”