Learning to Rank for Information Retrieval and Natural Language Processing

By Hang Li. 2011

406 citations recorded by the source corpus.

In this tutorial I will introduce 'learning to rank', a machine learning technology on constructing a model for ranking objects using training data. I will first explain the problem formulation of learning to rank, and relations between learning to rank and the other learning tasks. I will then describe learning to rank methods developed in recent years, including pointwise, pairwise, and listwise approaches. I will then give an introduction to the theoretical work on learning to rank and the applications of learning to rank. Finally, I will show some future directions of research on learning to rank. The goal of this tutorial is to give the audience a comprehensive survey to the technology and stimulate more research on the technology and application of the technology to natural language processing.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, neutral): “The skepticism expressed by Lin [2018] was retracted in short order [Lin, 2019a], as many researchers quickly demonstrated that with pretrained transformer models, large amounts of relevance judgments were not necessary to build effective models for text ranking.”
Semantic Matching in Search (2014, unknown): “In fact, machine learning techniques for the purpose, referred to as learning to rank, have been intensively studied and widely applied in web search [128, 115].”
Online Learning: A Comprehensive Survey (2018, background): “Learning to rank is an important family of machine learning techniques for information retrieval (Trotman, 2005; Cao et al., 2007; Hang, 2011; Zoghi et al., 2017; Shah et al., 2017).”
Multi-Stage Document Ranking with BERT (2019, unknown): “Although machine-learned ranking models have been well studied since the mid-2000s under the banner of “learning to rank”, the paradigm is heavily driven by manual feature engineering (Liu, 2009; Li, 2011); commercial web search engines are known to incorporate thousands of features (or more) in the”
From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing (2018, unknown): “We employ hinge loss (max-margin loss function) that has beenwidely used in the learning to rank literature for pairwise models [25].”
Learning With Average Precision: Training Image Retrieval With a Listwise Loss (2019, background): “Image retrieval can indeed be seen as a learning to rank problem [6, 9, 34, 57].”
BEER: BEtter Evaluation as Ranking (2014, background): “The output of learning in this approach can be either a function that ranks all hypotheses directly (global ranking model) or a function that assigns a score to each hypothesis individually which can be used for ranking (local ranking model) (Li, 2011).”
A Deep Look into Neural Ranking Models for Information Retrieval (2019, background): “Many different ranking models have been proposed over the past decades, including vector space models [1], probabilistic models [2], and learning to rank (LTR) models [3, 4].”
Gradient descent optimization of smoothed information retrieval metrics (2010, methodology): “2000; Joachims 2002) uses ‘(t) = max(0, 1 - t), while RankNet (Burges et al. 2005) uses ‘ðtÞ 1⁄4 logð1þ expð tÞÞ:GBRank (Zheng et al.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “With the development of machine learning discipline, learning to rank [19], [20] introduces supervised learning for text ranking.”
Training Efficient Tree-Based Models for Document Ranking (2013, background): “In particular, gradient-boosted regression trees (GBRTs) have proven to be highly effective, as documented in both the academic literature [1, 2] and in production commercial search engines such as Bing [3].”
Learning to Rank Using User Clicks and Visual Features for Image Retrieval (2015, unknown): “questions, and summarizing documents [1].”
From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing (2018, unknown): “We employ hinge loss (max-margin loss function) that has beenwidely used in the learning to rank literature for pairwise models [25].”
Interactive machine learning for health informatics: when do we need the human-in-the-loop? (2016, methodology): “PL can be seen as a natural link between ML and decision support, and was primarily applied in information retrieval with the central task of learning to rank [56, 57].”
Learning to rank for information retrieval (2009, unknown): “Representative algorithms include [4, 41, 42, 43, 44, 45, 46, 121, 134].”
Effectiveness/efficiency tradeoffs for candidate generation in multi-stage retrieval architectures (2013, background): “Categories and Subject Descriptors: H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval General Terms: Algorithms, Experimentation Keywords: query evaluation; postings intersection”
Document Ranking with a Pretrained Sequence-to-Sequence Model (2020, unknown): “Prior to deep learning, researchers and practitioners mostly adopt the paradigm known as “learning to rank”, which is heavily driven by manual feature engineering (Liu, 2009; Li, 2011).”