Towards Robust Neural Retrieval Models with Synthetic Pre-Training

By Revanth Reddy Gangi Reddy, Vikas Yadav, Md Arafat Sultan, Martin Franz, Vittorio Castelli, Heng Ji, Avirup Sil. 2021

15 citations recorded by the source corpus.

Recent work has shown that commonly available machine reading comprehension (MRC) datasets can be used to train high-performance neural information retrieval (IR) systems. However, the evaluation of neural IR has so far been limited to standard supervised learning settings, where they have outperformed traditional term matching baselines. We conduct in-domain and out-of-domain evaluations of neural IR, and seek to improve its robustness across different scenarios, including zero-shot settings. We show that synthetic training examples generated using a sequence-to-sequence generator can be effective towards this goal: in our experiments, pre-training with synthetic examples improves retrieval performance in both in-domain and out-of-domain evaluation on five different test sets.

Subfields: Artificial Intelligence, Retrieval-Augmented Generation, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Retrieval-Augmented Generation

In Information Retrieval & Search

Citations in the corpus

Learning to Retrieve Passages without Supervision (2021, methodology): “These datasets are expensive and sometimes even impractical to collect (e.g., for new languages or domains), and models trained on them often fail to generalize to new question distributions (Sci-avolino et al., 2021; Reddy et al., 2021).”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “[234] further extend this approach by incorporating an additional selection step for enhancing the generated question-text pairs for retrieval tasks.”