Truth Finding on the Deep Web: Is the Problem Solved?

By Xian Li, Xin Luna Dong, Kenneth Lyons, Weiyi Meng, Divesh Srivastava. 2012

317 citations recorded by the source corpus.

The amount of useful information available on the Web has been growing at a dramatic pace in recent years and people rely more and more on the Web to fulfill their information needs. In this paper, we study truthfulness of Deep Web data in two domains where we believed data are fairly clean and data quality is important to people's lives: Stock and Flight. To our surprise, we observed a large amount of inconsistency on data from different sources and also some sources with quite low accuracy. We further applied on these two data sets state-of-the-art data fusion methods that aim at resolving conflicts and finding the truth, analyzed their strengths and limitations, and suggested promising research directions. We wish our study can increase awareness of the seriousness of conflicting data on the Web and in turn inspire more research in our community to tackle this problem.

Subfields: Artificial Intelligence, Computer Science Applications, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Management Science and Operations Research

Citations in the corpus

BoostClean: Automated Error Detection and Repair for Machine Learning (2017, background): “Such labels may be collected as part of a gold standard dataset [36] or by cross-referencing the data with other sources [34].”
A Survey on Truth Discovery (2015, methodology): “In the light of this challenge, the topic of truth discovery [13,19,20,25–27,32,34,53,56,57] has gained increasing popularity recently due to its ability to estimate source reliability degrees and infer true information.”
Big data integration (2013, background): “[32], a study of 55 sources in the stock market domain identified 153 global attributes that are manually matched from 333 local attributes.”
SLiMFast: Guaranteed Results for Data Fusion and Source Reliability (2015, background): “This is a popular data fusion dataset [24] where sources provide information on the volume of stocks, i.”