Selective Weak Supervision for Neural Information Retrieval

By Kaitao Zhang, Chenyan Xiong, Zhenghao Liu, Zhiyuan Liu. 2020

46 citations recorded by the source corpus.

This paper democratizes neural information retrieval to scenarios where large scale relevance training signals are not available. We revisit the classic IR intuition that anchor-document relations approximate query-document relevance and propose a reinforcement weak supervision selection method, ReInfoSelect, which learns to select anchor-document pairs that best weakly supervise the neural ranker (action), using the ranking performance on a handful of relevance labels as the reward. Iteratively, for a batch of anchor-document pairs, ReInfoSelect back propagates the gradients through the neural ranker, gathers its NDCG reward, and optimizes the data selection network using policy gradients, until the neural ranker’s performance peaks on target relevance metrics (convergence). In our experiments on three TREC benchmarks, neural rankers trained by ReInfoSelect, with only publicly available anchor data, significantly outperform feature-based learning to rank methods and match the effectiveness of neural rankers trained with private commercial search logs. Our analyses show that ReInfoSelect effectively selects weak supervision signals based on the stage of the neural ranker training, and intuitively picks anchor-document pairs similar to query-document pairs.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Relevance-guided Supervision for OpenQA with ColBERT (2020, background): “…BM25, and existing inner-loop retrieval strategies restrict training the document encoder or require frequent re-indexing and repeated retrieval, RGS offers a scal-2 (Zhang et al., 2020) and headings text (MacAvaney et al., 2019), treated as queries for which the target content is assumed relevant.”
Long Document Ranking with Query-Directed Sparse Transformer (2020, methodology): “We then evaluate the generalization ability of QDS-Transformer in the few-shot setting (Zhang et al., 2020) using TREC datasets Robust04 (RB04), ClueWeb09-B (CW09), and ClueWeb12-B13 (CW12), in which labels are much fewer than DL track.”
MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale (2020, result): “Similar approaches were also explored in ad-hoc retrieval (Zhang et al., 2020; Ma et al., 2020; MacAvaney et al., 2019).”