Detecting spam web pages through content analysis

By Alexandros Ntoulas, Marc Najork, Mark S. Manasse, Dennis Fetterly. The Web Conference, 2006

737 citations recorded by the source corpus.

In this paper, we continue our investigations of "web spam": the injection of artificially-created pages into the web in order to influence the results from search engines, to drive traffic to certain pages for fun or profit. This paper considers some previously-undescribed techniques for automatically detecting spam pages, examines the effectiveness of these techniques in isolation and when aggregated using classification algorithms. When combined, our heuristics correctly identify 2,037 (86.2%) of the 2,364 spam pages (13.8%) in our judged collection of 17,168 pages, while misidentifying 526 spam and non-spam pages (3.1%).

Subfields: Artificial Intelligence, Information Systems, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/df3999e457a95d79846b55c2b92b211b39290b19

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. B. Liu
2. Leman Akoglu
3. Bing Liu
4. Yejin Choi
5. Claire Cardie
6. Myle Ott
7. Arjun Mukherjee
8. Eduard H. Hovy

In Information Systems

1. Leman Akoglu
2. B. Liu
3. Arjun Mukherjee
4. Claire Cardie
5. Myle Ott
6. Reza Zafarani
7. Bing Liu
8. Yejin Choi

In Sociology and Political Science

1. Leman Akoglu
2. Reza Zafarani
3. Xinyi Zhou
4. B. Liu
5. Arjun Mukherjee
6. Myle Ott
7. Claire Cardie
8. Yejin Choi

Citations in the corpus

• A Survey of Fake News (2020, methodology): “…content-based algorithms, which analyze web content features, such as word counts and content duplication [Baly et al. 2018; Fetterly et al. 2005; Ntoulas et al. 2006]; (ii) link-based algorithms, which detect web spam by utilizing graph information [Horne et al. 2019; Zhou and Pei 2009],…”
• Spotting fake reviewer groups in consumer reviews (2012, background): “[28] Martinez-Romo, J. and Araujo, A. Web Spam Identification Through Language Model Analysis.”
• Seven Months with the Devils: A Long-Term Study of Content Polluters on Twitter (2011, methodology): “Others are spam email analysis based on data compression algorithms (Bratko et al. 2006), machine learning (Goodman , Heckerman, and Rounthwaite 2005; Sahami et al. 1998) or statistics (Fetterly, Manasse, and Najork 2004; Ntoulas et al. 2006; Yoshida et al. 2004).”
• Fake News: A Survey of Research, Detection Methods, and Opportunities (2018, methodology): “ck spams. Algorithms to detect web spam thus can be classified into (i) content-based algorithms, which analyze web content features, such as word counts and content duplication [Fetterly et al. 2005;Ntoulas et al. 2006]; (ii) link-based algorithms, which detect web spam by utilizing graph informati”
• Detecting Fake Content with Relative Entropy Scoring (2008, background): “Based on such attributes, it is fairly straightforward to build robust, genre independent, classification systems that can sort salads from natural texts with a pretty high accuracy [5, 13, 11].”
• Opinion spam and analysis (2008, background): “INTRODUCTION The Web has dramatically changed the way that people express themselves and interact with others.”
• Review spam detection (2007, background): “For example, if one wants to buy a product, one typically goes to a merchant site (e.g., amazon.com) to read some Copyright is held by the author/owner(s).”
• Graph based anomaly detection and description: a survey (2014, background): “…features are complementary to other techniques that use content-based features, such as the number of words, number of hyperlinks, text redundancy, etc. [Canali et al., 2011, Ntoulas et al., 2006], and content-free features, such as URLbased-only host and lexical features [Ma et al., 2009].”
• On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection (2018, background): “Deceptive reviews can also be seen as an instance of spamming and online fraud [2, 16, 27, 56].”
• Shady paths: leveraging surfing crowds to detect malicious web pages (2013, methodology): “…Invasive software; H.3.5 [Online Information Services]: Web-based services General Terms Security Keywords HTTP redirections, detection, malware Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are…”
• PREDATOR: Proactive Recognition and Elimination of Domain Abuse at Time-Of-Registration (2016, background): “The detection can be based on the page content [43, 57], the presence of cloaking and redirection [35, 62], or the link structure leading to the pages [64].”
• Towards a General Rule for Identifying Deceptive Opinion Spam (2014, methodology): “Spam has been historically studied in the contexts of Web text (Gy¨ongyi et al., 2004; Ntoulas et al., 2006) or email (Drucker et al., 1999).”
• To Clean or Not to Clean (2018, background): “A great deal of work has been done on IR in the presence of adversarial information providers and on detecting spam pages [12, 13, 21, 39, 53, 67].”
• Finding Deceptive Opinion Spam by Any Stretch of the Imagination (2011, background): “Spam has historically been studied in the contexts of e-mail (Drucker et al., 2002), and the Web (Gy¨ongyi et al., 2004; Ntoulas et al., 2006).”