TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval

By Wenhao Lu, Jian Jiao, Ruofei Zhang. 2020

55 citations recorded by the source corpus.

Pre-trained language models like BERT have achieved great success in a wide variety of NLP tasks, while the superior performance comes with high demand in computational resources, which hinders the application in low-latency IR systems. We present TwinBERT model for effective and efficient retrieval, which has twin-structured BERT-like encoders to represent query and document respectively and a crossing layer to combine the embeddings and produce a similarity score. Different from BERT, where the two input sentences are concatenated and encoded together, TwinBERT decouples them during encoding and produces the embeddings for query and document independently, which allows document embeddings to be pre-computed offline and cached in memory. Thereupon, the computation left for run-time is from the query encoding and query-document crossing only. This single change can save large amount of computation time and resources, and therefore significantly improve serving efficiency. Moreover, a few well-designed network layers and training strategies are proposed to further reduce computational cost while at the same time keep the performance as remarkable as BERT model. Lastly, we develop two versions of TwinBERT for retrieval and relevance tasks correspondingly, and both of them achieve close or on-par performance to BERT-Base model. The model was trained following the teacher-student framework and evaluated with data from one of the major search engines. Experimental results showed that the inference time was significantly reduced and was firstly controlled around 20ms on CPUs while at the same time the performance gain from fine-tuned BERT-Base model was mostly retained. Integration of the models into production systems also demonstrated remarkable improvements on relevance metrics with negligible influence on latency.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Pre-trained models for natural language processing: A survey (2020, methodology): “Distillation from soft target probabilities can also be used in task-specific models, such as information retrieval [115], and sequence labeling [175].”
Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling (2021, methodology): “2 BERTDOT Dense Retrieval Model The BERTDOT model encodes query q1:m and passage p1:n sequences independently from each other and matches only a single representation vector of the query with a single representation vector of a passage [25, 26, 42].”
Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction (2022, background): “2 BERTDOT and ColBERT Architectures BERTDOT matches a single vector of the query with a single vector of a passage, produced by independent BERT computations [30, 32, 55].”
Deep Learning--based Text Classification (2020, methodology): “As BERT becomes the new state of the art sentence embedding model, there have been attempts to building BERT-based S2Nets, such as SBERT [124] and TwinBERT [125].”
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation (2020, methodology): “KD has been applied in the ranking task for the same architecture with fewer layers [5, 13, 24] and in constrained sub-tasks, such as keyword-list matching [25].”