A Confidence-Aware Approach for Truth Discovery on Long-Tail Data

By Qi Li, Yaliang Li, Jing Gao, Lu Su, Bo Zhao, Murat Demirbas, Wei Fan, Jiawei Han. Proceedings of the VLDB Endowment, 2014

346 citations recorded by the source corpus.

In many real world applications, the same item may be described by multiple sources. As a consequence, conflicts among these sources are inevitable, which leads to an important task: how to identify which piece of information is trustworthy, i.e., the truth discovery task. Intuitively, if the piece of information is from a reliable source, then it is more trustworthy, and the source that provides trustworthy information is more reliable. Based on this principle, truth discovery approaches have been proposed to infer source reliability degrees and the most trustworthy information (i.e., the truth) simultaneously. However, existing approaches overlook the ubiquitous long-tail phenomenon in the tasks, i.e., most sources only provide a few claims and only a few sources make plenty of claims, which causes the source reliability estimation for small sources to be unreasonable. To tackle this challenge, we propose a confidence-aware truth discovery (CATD) method to automatically detect truths from conflicting data with long-tail phenomenon. The proposed method not only estimates source reliability, but also considers the confidence interval of the estimation, so that it can effectively reflect real source reliability for sources with various levels of participation. Experiments on four real world tasks as well as simulated multi-source long-tail datasets demonstrate that the proposed method outperforms existing state-of-the-art truth discovery approaches by successful discounting the effect of small sources.

Subfields: Artificial Intelligence, Computer Science Applications

View Paper: https://www.semanticscholar.org/paper/cd48760a142830b796b8a85a158cd469f3e5feb0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

SURF: improving classifiers in production by learning from busy and noisy end users (2020, methodology): “We will compare SURF to two groups of crowdsourcing algorithms: GROUP 1: Dawid-Skene (DS) [1], iBCC [3], EBCC [6], LFC [8]) and GROUP 2: ZenCrowd (ZC) [2], CATD [5], GLAD [12], Majority Vote (MV) ) For each instantiation of the parameters we vary we run the experiments 10 times, resampling N = 1000 ”
A Survey on Truth Discovery (2015, background): “To better deal with such “small” sources, in [25], the authors propose a conﬁdence-aware truth discovery approach that outputs the conﬁdence interval of the source reliability estimation.”
SLiMFast: Guaranteed Results for Data Fusion and Source Reliability (2015, background): “…collaborators from a medical school—who are currently engaged in extracting information from scientific articles to populate a structured data repository—we find that using data fusion can be challenging for users despite the numerous approaches proposed in the literature [9, 14, 21, 22, 28, 38].”
Toward Annotator Group Bias in Crowdsourcing (2021, neutral): “…et al. (2012); Aydin et al. (2014) use worker probability to model the ability of an annotator to correctly answer a task, and some other works (Whitehill et al., 2009; Li et al., 2014b) introduce a similar concept, worker quality, by changing the value range from [0 , 1] to ( −∞ , + ∞ ) .”
DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning (2018, methodology): “Truth discovery: Prior approaches for truth discovery (Yin et al., 2008; Dong et al., 2009, 2015; Li et al., 2011, 2014, 2015; Pasternack and Roth, 2011, 2013; Ma et al., 2015; Zhi et al., 2015; Gao et al., 2015; Lyu et al., 2017) have focused on structured data with the goal of addressing the…”