Introducing LETOR 4.0 Datasets

By Tao Qin, Tie-Yan Liu. 2013

458 citations recorded by the source corpus.

LETOR is a package of benchmark data sets for research on LEarning TO Rank, which contains standard features, relevance judgments, data partitioning, evaluation tools, and several baselines. Version 1.0 was released in April 2007. Version 2.0 was released in Dec. 2007. Version 3.0 was released in Dec. 2008. This version, 4.0, was released in July 2009. Very different from previous versions (V3.0 is an update based on V2.0 and V2.0 is an update based on V1.0), LETOR4.0 is a totally new release. It uses the Gov2 web page collection (~25M pages) and two query sets from Million Query track of TREC 2007 and TREC 2008. We call the two query sets MQ2007 and MQ2008 for short. There are about 1700 queries in MQ2007 with labeled documents and about 800 queries in MQ2008 with labeled documents. If you have any questions or suggestions about the datasets, please kindly email us (letor@microsoft.com). Our goal is to make the dataset reliable and useful for the community.

Subfields: Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Control and Systems Engineering

Citations in the corpus

Off-policy evaluation for slate recommendation (2016, methodology): “0 MQ2008 dataset [20] to create a contextual bandit instance.”
Tabular Data: Deep Learning is Not All You Need (2021, background): “…are Forest Cover Type, Higgs Boson, and Year Prediction [Dua and Graff, 2017], Rossmann Store Sales [Kaggle, 2019a], Gas Concentrations, Eye Movements, and Gesture Phase [Vanschoren et al., 2014], MSLR [Qin and Liu, 2013], Epsilon [Pascal, 2008], Shrutime [Kaggle, 2019b], and Blastchar [IBM, 2019].”
MergeDTS (2018, methodology): “First, to answer RQ1 through RQ3 and RQ6, we run experiments on three large-scale datasets: the Microsoft Learning to Rank (MSLR) WEB30K dataset [39], the Yahoo! Learning to Rank Challenge Set 1 (Yahoo) [10], and the ISTELLA dataset [35].”
Posthoc Interpretability of Learning to Rank Models using Secondary Training Data (2018, methodology): “ results. 3 EXPERIMENTAL SETUP TobestanswertheresearchquestionsposedinSection1,wedevised the following experiemtal setup: All data used for the experiemts was derived from the learningto-rank dataset [7] released by Microso›. We use the dataset consisting of 30k web search queries. Each query has be”
Leveraging Passage-level Cumulative Gain for Document Ranking (2020, methodology): “We extract eight learning-to-rank features for each passage according to Qin and Liu [33], including the passage length (the number of words in the passage), the average TF, IDF, and TF × IDF values of query terms in the passage, scores of BM25 and three language models with the query.”
A General Framework for Counterfactual Learning-to-Rank (2018, methodology): “g the generalized counterfactual LTR approach. 5.1 Setup We conducted experiments on synthetic click data derived from two major LTR datasets, the Yahoo Learning to Rank Challenge corpus and LETOR4.0 [22]. LETOR4.0 contains two separate corpora: MQ2007 and MQ2008. Since MQ2008 is significantly small”
LightGBM: A Highly Efficient Gradient Boosting Decision Tree (2017, methodology): “Among them, the Microsoft Learning to Rank (LETOR) [26] dataset contains 30K web search queries.”
Human Behavior Inspired Machine Reading Comprehension (2019, unknown): “Liu et al. [20] proposed a two-stage examination model on the examination behavior on search engine result pages (SERPs) and found that there usually exists a skim-ming step before the user really reads the search result, which can help estimate better relevance of search results.”
Revisiting Deep Learning Models for Tabular Data (2021, background): “The datasets include: California Housing (CA, real estate data, Kelley Pace and Barry (1997)), Adult (AD, income estimation, Kohavi (1996)), Helena (HE, anonymized dataset, 2013)).”
PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2020, background): “And for MQ2007 and MQ2008 datasets, we follow the data partition in LETOR4.”
On Embeddings for Numerical Features in Tabular Deep Learning (2022, unknown): “We have also shown that traditional MLP-like models coupled with embeddings can perform on par with attention-based models.”
TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank (2018, background): “First, they were developed for small labeled data sets such as LETOR [21] (thousands of examples), but not for massive click log datasets (hundreds of millions of examples) that are common in industrial applications.”
A Survey of Model Architectures in Information Retrieval (2025, unknown): “The success of LTR methods can be largely attributed to their extensive use of manually engineered features, derived from both statistical properties of text terms and user behavior data collected from web browsing traffic (Qin and Liu, 2013).”