Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring

By Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, Jason Aaron Edward Weston. 2019

319 citations recorded by the source corpus.

The use of deep pre-trained transformers has led to remarkable progress in a number of applications (Devlin et al., 2018). For tasks that make pairwise comparisons between sequences, matching a given input with a corresponding label, two approaches are common: Cross-encoders performing full self-attention over the pair and Bi-encoders encoding the pair separately. The former often performs better, but is too slow for practical use. In this work, we develop a new transformer architecture, the Poly-encoder, that learns global rather than token level self-attention features. We perform a detailed comparison of all three approaches, including what pre-training and fine-tuning strategies work best. We show our models achieve state-of-the-art results on four tasks; that Poly-encoders are faster than Cross-encoders and more accurate than Bi-encoders; and that the best results are obtained by pre-training on large datasets similar to the downstream tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Dense Passage Retrieval for Open-Domain Question Answering (2020, methodology): “With the help of FAISS in-memory index for real-valued vectors 9 , DPR can be made incredibly efﬁcient, processing 995.0 questions per second, returning top 100 passages per question.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “Our research on more efficient models (Humeau et al., 2019) is a step in that direction, and so are works that explore smaller footprints for Transformer-based models, e.”
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, background): “Their efﬁciency also becomes a concern when pretrained models are used [37, 38].”
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, methodology): “Following Humeau et al. (2019), this is a previously existing Reddit dataset extracted and obtained by a third party and made available on pushshift.io (Baum-gartner et al., 2020), spanning 1.5B training examples from Reddit obtained from PushShift 3 through July 2019.”
Retrieval Augmentation Reduces Hallucination in Conversation (2021, methodology): “However, allowing more interaction between a context and candidate yields superior results in various information retrieval and ranking tasks (Humeau et al., 2020; Khattab and Zaharia, 2020).”
Learning Dense Representations of Phrases at Scale (2020, methodology): “This type of dense retrieval has been also studied for sentence and passage retrieval (Humeau et al., 2019; Karpukhin et al., 2020) (see Lin et al., 2020 for recent advances in dense retrieval).”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “’s original arXiv paper did not appear in a peer-reviewed venue until April 2020, at ICLR [Humeau et al., 2020].”
Autoregressive Entity Retrieval (2020, background): “, 2020), the dot-product can miss fine-grained interactions between input and entity meta information (Humeau et al., 2020).”
Recipes for Building an Open-Domain Chatbot (2020, methodology): “Figure 2: The Poly-encoder Transformer architecture (Humeau et al., 2019) for retrieval encodes global features of the context using multiple representations (codes), which are attended to by each possible candidate response.”
Hash Layers For Large Sparse Models (2021, methodology): “Following [32], we use a previously existing Reddit dataset extracted and obtained by a third party and made available on pushshift.”
Controlling Style in Generated Dialogue (2020, methodology): “The retriever model is a 660M-parameter Poly-encoder, consisting of two Transformer encoders for context strings and candidate responses, whose outputs are attended over to produce a ranking of candidates (Humeau et al., 2020).”
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers (2021, methodology): “Recent chit-chat with the neural model, like Meena and Blender, shows impressive conversational performance (Humeau et al., 2020; Adiwardana et al., 2020; Roller et al., 2020).”
Deploying Lifelong Open-Domain Dialogue Learning (2020, methodology): “In particular, we use the Poly-Encoder (PE) Transformer architecture as a base (Humeau et al., 2019), as it provides state of the art results compared to other retrieval models, whilst being tractable to deploy.”
Augmenting Transformers with KNN-Based Composite Memory for Dialog (2020, background): “This often leverages self-attention or cross-attention mechanisms (Humeau et al., 2019).”
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021, methodology): “There, pretrained language models (LMs) such as BERT [12] or RoBERTa [36] are fine-tuned via twin-network architectures on auxiliary tasks such as semantic textual similarity [45, 25], paraphrasing [55], response retrieval [57, 22, 21, 25], or translation ranking [10, 16].”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “The poly-encoder model [91] learns m different representations for modeling the semantics of a text in multiple views, called context codes in the original paper.”
Situated Dialogue Learning through Procedural Environment Generation (2021, methodology): “Adaptive Pre-training is done on the tasks introduced in Ammanabrolu et al. (2021) by training a 12 layer transformer with 256 million parameters using a cross-entropy loss as seen in Humeau et al. (2020).”
How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds (2020, methodology): “Pre-training is done on the tasks described in Section 4.3 by training a 12 layer transformer with 256 million parameters using a cross-entropy loss as seen in (Humeau et al., 2020).”
ReTraCk: A Flexible and Efficient Framework for Knowledge Base Question Answering (2021, background): “Schema Retriever As schema items are not always mentioned explicitly in the question and their vocabularies are much fewer than entities 13 , we leverage the dense retriever framework (Mazar´e et al., 2018; Humeau et al., 2020; Wu et al., 2020) to obtain the related types and relations.”
Boosted Dense Retriever (2021, background): “Humeau et al. (2020) represent queries with multiple vectors, but retrieval is comparatively slow as relevance cannot be calculated with a single MIPS call.”