Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring

By Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, Jason Aaron Edward Weston. 2019

248 citations recorded by the source corpus.

Subfields: Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Model Architectures

Citations in the corpus

Distilling Knowledge from Reader to Retriever for Question Answering (2020, background): “To address this limitation, Humeau et al. (2019) introduced the poly-encoder architecture, in which each documents is represented by multiple vectors instead of one.”
Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “To alleviate the limitations of bi-encoders, Humeau et al. (2019) introduces the poly-encoder architecture, where documents are encoded by multiple vectors.”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “owledge, the ﬁrst to have proposed successful neural architectures for ranking using transformer-based dense representations, pre-dating Sentence-BERT [Reimers and Gurevych, 2019] (Section 4.4); cf. [Humeau et al., 2019]. They introduced the bi-encoder and cross-encoder terminology that we have adop”
Autoregressive Entity Retrieval (2020, methodology): “First, unless a costly cross-encoder is used for re-ranking (Wu et al., 2019), the dot-product can miss ﬁne-grained interactions between input and entity meta information (Humeau et al., 2019).”
Sparse, Dense, and Attentional Representations for Text Retrieval (2020, methodology): “This efficiency is a key difference from the POLY-ENCODER (Humeau et al., 2020), which computes a fixed number of vectors per query, and aggregates them by softmax attention against document vectors.”
Multi-Dimensional Gender Bias Classification (2020, background): “Classes are then scored (and ranked) by taking a dot product between the representations of the textual input and a given class, following the bi-encoder architecture (Humeau et al., 2019) trained with cross entropy.”
PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them (2021, methodology): “The reranker uses cross-encoding (Humeau et al., 2020), and includes the retrieved answer in the scoring function for richer featurisation.”
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents (2019, methodology): “We use BPE following Humeau et al. (2019) via lower-cased Wikipedia, Toronto Books, and Open Subtitles with 30k merges, giving 54,940 terms.”
Pre-training Tasks for Embedding-based Large-scale Retrieval (2020, background): “Humeau et al. (2019) propose the Poly-encoders architecture to balance the computation/expressiveness tradeoff between two-tower models and cross-attention models.”
Zero-shot Entity Linking with Dense Entity Retrieval (2019, methodology): “Architecture We use a bi-encoder architecture similar to the work of (Humeau et al., 2019) to model (mention, entity) pairs.”
Generating Fact Checking Briefs (2020, methodology): “BLINK combines a bi-encoder (Urbanek et al., 2019; Humeau et al., 2019) that identifies candidates with a cross-encoder that models the interaction between mention context and en-”
Domain-matched Pre-training Tasks for Dense Retrieval (2021, methodology): “This dataset was originally extracted and made available by pushshift.io and shown to be useful for dialogue and chit-chat applications previously (Humeau et al., 2019; Roller et al., 2020).”
Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training (2019, methodology): “Following previous work (Humeau et al., 2019), we pre-train our model on dialogue data, using a previously existing Reddit dataset extracted and obtained by a third party and made available on pushshift.io, training to generate a comment conditioned on the full thread leading up to the comment,…”
Scalable Zero-shot Entity Linking with Dense Entity Retrieval (2019, methodology): “Architecture We use a bi-encoder architecture similar to the work of Humeau et al. (2019) to model (mention, entity) pairs.”
BookSum: A Collection of Datasets for Long-form Narrative Summarization (2021, background): “Cross-Encoder (Humeau et al., 2019) performs self-attention over the sentence and paragraph text passed together through a Transformer network to generate a similarity score between the input pair.”
I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling (2020, methodology): “These include Poly-encoder (Humeau et al., 2019) based retrieval models, generative models (Roller et al., 2020), unlikelihood trained models (Li et al., 2019a), retrieve-and-reﬁne models (Weston et al., 2018; Roller et al., 2020), models either pre-trained on a previously existing Red-dit dataset e”
Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction (2022, background): “[18] compute a fixed number of vectors per query, and aggregate them by softmax attention against document vectors.”
ConveRT: Efficient and Accurate Conversational Representations from Transformers (2019, methodology): “Their dual encoder (i.e., bi-encoder ) is based on the BERT-base architecture (Humeau et al., 2020): it relies on 12 Transformer blocks, 12 attention heads, and a hidden size dimensionality of 768 (while we use 512).”
Empirical study on BlenderBot 2.0’s Errors Analysis in terms of Model, Data and User-Centric Approach (2022, methodology): “Thereby, BlenderBot 2.0 (Xu et al., 2021; Komeili et al., 2021), which is an improved version of BlenderBot 1.0 was developed.”
All-in-One Image-Grounded Conversational Agents (2019, methodology): “[Devlin et al., 2019]; and starting from two versions of pushshift.io Reddit training, i.e., Transformers with 79M parameters from [Mazaré et al., 2018] and 128M parameters from [Humeau et al., 2019].”