Real-time Inference in Multi-sentence Tasks with Deep Pretrained Transformers

By Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, Jason Aaron Edward Weston. 2019

42 citations recorded by the source corpus.

Subfields: Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Dense Passage Retrieval for Open-Domain Question Answering (2020, background): “In contrast to unsupervised dimensionality reduction methods (e.g., LSA), leveraging labeled data to train dense encoders for embedding both documents and queries has become popular recently (Yih et al., 2011; Huang et al., 2013; Gillick et al., 2019; Humeau et al., 2019).”
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons (2019, neutral): “Finally, we also compare to the Polyencoder model ( PE , Humeau et al., 2019), a recent state-of-the-art retrieval model.”
Autoregressive Entity Retrieval (2020, methodology): “First, unless a costly cross-encoder is used for re-ranking (Wu et al., 2019), the dot-product can miss ﬁne-grained interactions between input and entity meta information (Humeau et al., 2019).”
Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, methodology): “Following Humeau et al. (2019), the model was pre-trained”
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019, methodology): “Humeau et al. (2019) addresses the run-time overhead of the cross-encoder from BERT and present a method (poly-encoders) to compute a score between m context vectors and pre-3 computed candidate embeddings using attention.”
Can You Put it All Together: Evaluating Conversational Agents’ Ability to Blend Skills (2020, methodology): “The base architecture used throughout the paper is the 256-million parameter poly-encoder proposed in Humeau et al. (2019), which is a Transformer-based architecture for retrieval that learns a small number of codes representing the input context, so that performing attention over retrieval…”
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents (2019, methodology): “Following (Humeau et al., 2019), we use a previously existing Reddit dataset extracted and obtained by a third party and made available on pushshift.”
Pre-training Tasks for Embedding-based Large-scale Retrieval (2020, background): “Humeau et al. (2019) propose the Poly-encoders architecture to balance the computation/expressiveness tradeoff between two-tower models and cross-attention models.”
Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training (2019, methodology): “Following previous work (Humeau et al., 2019), we pre-train our model on dialogue data, using a previously existing Reddit dataset extracted and obtained by a third party and made available on pushshift.io, training to generate a comment conditioned on the full thread leading up to the comment,…”
Augmenting Transformers with KNN-Based Composite Memory for Dialog (2020, positive): “This often leverages self-attention or cross-attention mechanisms (Humeau et al., 2019).”
Hierarchical Inductive Transfer for Continual Dialogue Learning (2022, methodology): “encoder with 256M parameters (Humeau et al., 2019) as the underlying model, pretrain it on the pushshift.”
Generating Interactive Worlds with Text (2019, methodology): “We use the BERT-based models proposed in (Urbanek et al. 2019; Humeau et al. 2019) to encode the location information and the negative candidates.”
All-in-One Image-Grounded Conversational Agents (2019, methodology): “…it has been shown that pre-training on utterances from large-scale conversational corpora such as from pushshift.io Reddit improves over large pre-training over resources like Wikipedia because they are more related to the task [Mazaré et al., 2018; Humeau et al., 2019; Shuster et al., 2019b].”
BART for Knowledge Grounded Conversations (2020, methodology): “The Wizard Generative Model [11] uses a Poly-encoder [5] to retrieve a single sentence from an external knowledge source.”