ConveRT: Efficient and Accurate Conversational Representations from Transformers

By Matthew Henderson, Iñigo Casanueva, Nikola Mrkvsi'c, Pei-hao Su, Tsung-Hsien, Ivan Vulic. 2019

214 citations recorded by the source corpus.

General-purpose pretrained sentence encoders such as BERT are not ideal for real-world conversational AI applications; they are computationally heavy, slow, and expensive to train. We propose ConveRT (Conversational Representations from Transformers), a pretraining framework for conversational tasks satisfying all the following requirements: it is effective, affordable, and quick to train. We pretrain using a retrieval-based response selection task, effectively leveraging quantization and subword-level parameterization in the dual encoder to build a lightweight memory- and energy-efficient model. We show that ConveRT achieves state-of-the-art performance across widely established response selection tasks. We also demonstrate that the use of extended dialog history as context yields further performance gains. Finally, we show that pretrained representations from the proposed encoder can be transferred to the intent classification task, yielding strong results across three diverse data sets. ConveRT trains substantially faster than standard sentence encoders or previous state-of-the-art dual encoders. With its reduced size and superior performance, we believe this model promises wider portability and scalability for Conversational AI applications.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Challenges in Building Intelligent Open-domain Dialog Systems (2019, methodology): “This structure has been widely adopted in retrieval-based methods [46].”
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021, methodology): “There, pretrained language models (LMs) such as BERT [12] or RoBERTa [36] are fine-tuned via twin-network architectures on auxiliary tasks such as semantic textual similarity [45, 25], paraphrasing [55], response retrieval [57, 22, 21, 25], or translation ranking [10, 16].”
Dialogue State Tracking with a Language Model using Schema-Driven Prompting (2021, background): “, 2020) and task-oriented dialogue systems (Mehri et al., 2019; Wu et al., 2020; Henderson et al., 2020; Peng et al., 2020b).”
Distilling Knowledge for Fast Retrieval-based Chat-bots (2020, methodology): “To find the best response to a particular conversation’s chat history traditional text retrieval methods such as term frequency have proven to be insufficient [10], therefore the majority of modern research focuses on neural ranking approaches [4, 6, 10].”
A Wizard-of-Oz Interface and Persona-based Methodology for Collecting Health Counseling Dialog (2020, methodology): “While DS performance has improved through recent advances in neural language modeling [14,19], there are still no tools for practitioners to build statistical HDS through communication with patients.”
Efficient Intent Detection with Dual Sentence Encoders (2020, methodology): “1) As shown by Henderson et al. (2019b), pretraining on a general language-modeling (LM) objective for conversational tasks is less effective than conversational pretraining based on the response selection task and conversational data (Henderson et al., 2019c; Mehri et al., 2019).”
Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference (2020, unknown): “USE+ConveRT performs worse than the RoBERTa-based classiﬁer on the OOD detection task.”
SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval (2020, background): “Many attempts have been made to improve the ﬁrst-stage retrieval performance (Chidambaram et al., 2018; Seo et al., 2019; Henderson et al., 2019; Karpukhin et al., 2020; Chang et al., 2020).”
DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue (2020, background): “Task-oriented dialogue poses a unique set of challenges; dialogue is intrinsically goal-driven, multiturn and often informal/noisy (Henderson et al., 2019; Zhang et al., 2019b).”
Multilingual and Cross-Lingual Intent Detection from Spoken Data (2021, methodology): “Here, ﬁxed universal sentence encoders such as USE (Cer et al., 2018; Chidambaram et al., 2019) or ConveRT (Henderson et al., 2020) We keep all pretrained sentence encoders ﬁxed during intent detection training.”