Further Boosting BERT-based Models by Duplicating Existing Layers: Some Intriguing Phenomena inside BERT
By Wei-Tsung Kao, Tsung-Han Wu, Po-Han Chi, Chun-Cheng Hsieh, Hung-yi Lee. 2020
5 citations recorded by the source corpus.
Although Bidirectional Encoder Representations from Transformers (BERT) have achieved tremendous success in many natural language processing (NLP) tasks, it remains a black box, so much previous work has tried to lift the veil of BERT and understand the functionality of each layer. In this paper, we found that removing or duplicating most layers in BERT would not change their outputs. This fact remains true across a wide variety of BERT-based models. Based on this observation, we propose a quite simple method to boost the performance of BERT. By duplicating some layers in the BERT-based models to make it deeper (no extra training required in this step), they obtain better performance in the down-stream tasks after fine-tuning.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Anna Rogers
- Anna Rumshisky
- Olga Kovaleva
- Yonatan Belinkov
- Samuel R. Bowman
- Francesca Toni
- Piyawat Lertvittayakumjorn
- Hannaneh Hajishirzi
Citations in the corpus
- A Primer in BERTology: What We Know About How BERT Works (2020, background): “Depending on the task, some BERT heads/ layers are not only redundant (Kao et al., 2020), but also harmful to the downstream task performance.”