Training Curricula for Open Domain Answer Re-Ranking

By Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, Ophir Frieder. 2020

20 citations recorded by the source corpus.

In precision-oriented tasks like answer ranking, it is more important to rank many relevant answers highly than to retrieve all relevant answers. It follows that a good ranking strategy would be to learn how to identify the easiest correct answers first (i.e., assign a high ranking score to answers that have characteristics that usually indicate relevance, and a low ranking score to those with characteristics that do not), before incorporating more complex logic to handle difficult cases (e.g., semantic matching or reasoning). In this work, we apply this idea to the training of neural answer rankers using curriculum learning. We propose several heuristics to estimate the difficulty of a given training sample. We show that the proposed heuristics can be used to build a training curriculum that down-weights difficult samples early in the training process. As the training process progresses, our approach gradually shifts to weighting all samples equally, regardless of difficulty. We present a comprehensive evaluation of our proposed idea on three answer ranking datasets. Results show that our approach leads to superior performance of two leading neural ranking architectures, namely BERT and ConvKNRM, using both pointwise and pairwise losses. When applied to a BERT-based ranker, our method yields up to a 4% improvement in MRR and a 9% improvement in P@1 (compared to the model trained without a curriculum). This results in models that can achieve comparable performance to more expensive state-of-the-art techniques.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

In Question Answering & Reading Comprehension

Jimmy J. Lin

Citations in the corpus

Achieving Human Parity on Visual Question Answering (2021, background): “Recently, open-domain QA [53, 58, 78, 96, 97] has become one of the most popular topics in question answering.”
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation (2020, background): “MacAvaney et al. [28] adapt the training procedure for answer ranking by reordering the training samples and shifting samples to the beginning which are estimated to be easy.”
Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling (2021, background): “[26] adapt the training procedure by shifting samples to the beginning which are estimated to be easy.”