Efficient and effective spam filtering and re-ranking for large web datasets

By Gordon V. Cormack, Mark D. Smucker, C. Clarke. 2010

355 citations recorded by the source corpus.

The TREC 2009 web ad hoc and relevance feedback tasks used a new document collection, the ClueWeb09 dataset, which was crawled from the general web in early 2009. This dataset contains 1 billion web pages, a substantial fraction of which are spam—pages designed to deceive search engines so as to deliver an unwanted payload. We examine the effect of spam on the results of the TREC 2009 web ad hoc and relevance feedback tasks, which used the ClueWeb09 dataset. We show that a simple content-based classifier with minimal training is efficient enough to rank the “spamminess” of every page in the dataset using a standard personal computer in 48 hours, and effective enough to yield significant and substantive improvements in the fixed-cutoff precision (estP10) as well as rank measures (estR-Precision, StatMAP, MAP) of nearly all submitted runs. Moreover, using a set of “honeypot” queries the labeling of training data may be reduced to an entirely automatic process. The results of classical information retrieval methods are particularly enhanced by filtering—from among the worst to among the best.

Subfields: Information Systems, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Information Retrieval & Search

Citations in the corpus

A Deep Relevance Matching Model for Ad-hoc Retrieval (2016, neutral): “ing the Waterloo Fusion spam scores [3].”
Relevance-based Word Embedding (2017, methodology): “e spam ltering phase was done using theWaterloo spam scorer5 [6] with the threshold of 60%.”
Neural Ranking Models with Weak Supervision (2017, methodology): “In our experiments with this collection, we ltered out the spam documents using the Waterloo spam scorer3 [7] with the default threshold 70%.”
Context-Aware Document Term Weighting for Ad-Hoc Search (2020, methodology): “Spam was filtered using the Waterloo spam score [6] with a threshold of 60.”
Efficient Document Re-Ranking for Transformers by Precomputing Term Representations (2020, background): “[3] for ClueWeb09-B, the size can be reduced to about 34TB.”
Overview of Touché 2022: Argument Retrieval (2022, positive): “For each document (text passage) in the retrieval corpus the following 3 components were computed: one-sentence extractive summary us-ing LexRank [28], premises and claims were identified with TARGER [19], and spam scores were found in the Waterloo Spam Rankings dataset [21].”
From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing (2018, methodology): “The spamfiltering phasewas done using theWaterloo spam scorer1 [9] with the threshold of 60%.”
EsdRank: Connecting Query and Documents through External Semi-Structured Data (2015, methodology): “Spam filtering is important for ClueWeb09, thus we removed the 70% spammiest documents using Waterloo spam scores [8].”
PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2020, methodology): “Note that ClueWeb09-B is filtered to the set of documents with spam scores in the 60 percentile, using the Waterloo Fusion spam scores [9].”
Query Expansion with Freebase (2015, background): “The ground truth score for a candidate term is generated by its influence on retrieved documents, when used for expansion individually.”
Phishing Attacks: A Recent Comprehensive Study and a New Anatomy (2021, methodology): “Spam filtering techniques (Cormack et al., 2011) and statistical classifiers (Bergholz et al.”
Efficient and effective retrieval using selective pruning (2013, background): “spamminess [11], field lengths, 5 fraction of stopwords, table text [4]) QI Click feature (click count) 1 QD term dependence/proximity models (MRF [26], pBiL [29]) 2”
Scalable Modified Kneser-Ney Language Model Estimation (2013, methodology): “7 After spam filtering (Cormack et al., 2011), removing markup, selecting English, splitting sentences (Koehn, 2005), deduplicating, tokenizing (Koehn et al., 2007), and truecasing, 126 billion tokens remained.”
Semantic Matching by Non-Linear Word Transportation for Information Retrieval (2016, methodology): “Both GOV2 and ClueWeb-09-Cat-B are large Web collections, where ClueWeb-09-Cat-B is filtered to the set of documents with spam scores in the 60 percentile, using the Waterloo Fusion spam scores [8].”
Effectiveness/efficiency tradeoffs for candidate generation in multi-stage retrieval architectures (2013, methodology): “, PageRank, content quality score [7], etc.”
The Simplest Thing That Can Possibly Work: Pseudo-Relevance Feedback Using Text Classification (2019, background): “Cor-mack et al. [6] used pseudo-negative labels to train spam classifiers in a distantly-supervised manner.”
Adversarial Web Search (2011, background): “Gordon Cormack and collaborators at the University of Waterloo has built a classifier from honeypot queries, labels from the WEBSPAM-UK2006 and WEBSPAM-UK2007 collections, and a small set of hand-labeled data [56] and evaluated it in terms of the effect on retrieval.”