Measuring Retrieval Effectiveness Based on User Preference of Documents

By Yiyu Yao. 1995

Subfields: Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Retrieval & Search

Citations in the corpus

Human Preferences as Dueling Bandits (2022, background): “Proposals to base information retrieval evaluations on preference judgments date back to the early 1990s [15, 30, 37].”
Recommender Systems (2012, unknown): “Similar to Kendall’s Tau, the normalized distance-based performance measure (NDPM) was originally proposed by Yao [98] to compare two different weakly ordered rankings.”
Evaluation Measures Based on Preference Graphs (2021, background): “In 1995, Yao [31] essentially proposed the core idea of this paper, using weak orderings of items as the basis for measurement and then defining a distance function between rankings.”
Music recommendation and discovery in the long tail (2008, background): “1]), between the user’s classification for a set of documents and the system’s classification for the same documents (Yao, 1995).”
Learning to Order Things (1997, unknown): “Problems that involve ordering and ranking have been investigated in various elds such as decision theory, the social sciences, information retrieval and mathematical economics [5, 21, 8, 13, 26, 27, 14, 35].”
Optimizing search engines using clickthrough data (2002, methodology): “Furthermore, it is proportional to the measure of Yao [26] proposed for evaluating information retrieval systems.”
Evaluating collaborative filtering recommender systems (2004, methodology): “NDPM was used to evaluate the accuracy of the FAB recommender system [Balabanovı́c and Shoham 1997]. It was originally proposed by Yao [1995]. Yao developed NDPM theoretically, using an approach from decision and measurement theory.”
Evaluating Recommendation Systems (2011, background): “The Normalized Distance-based Performance Measure (NDPM) measure [61] is suitable for such cases.”
Fab: content-based, collaborative recommendation (1997, methodology): “For each point in time we then measured the distance between the users’ rankings and the rankings predicted from their profiles, using the ndpm measure as defined by Yao [ 10 ].”
Preference-based Evaluation Metrics for Web Image Search (2020, background): “Yao [29] defines “distance” between a given ranking and a ground-truth ranking to form a new measure of system performance named distance-based Performance Measure (Dpm).”
Collaborative filtering recommender systems (2013, methodology): “Ramax is the highest possible utility if the system ranked the products in the actual purchase that the individual rated them: The NDPM measure: NDPM was initially suggested by Yao (1995) and used to assess the precision of the FAB recommender system (Balabanovíc and Shoham, 1997).”
A Survey of Collaborative Filtering Techniques (2009, background): “[60], metrics evaluating recommendation systems can be broadly classified into the following broad categories: predictive accuracy metrics, such as Mean Absolute Error (MAE) and its variations; classification accuracy metrics, such as precision, recall, F1-measure, and ROC sensitivity; rank accuracy”
Using Topic Models in Content-Based News Recommender Systems (2013, methodology): “For the random baseline method, we can see average NDPM value very close to 0.5, the expected value for a random ranking.”