Discovery of Web Robot Sessions Based on their Navigational Patterns

By Pang-Ning Tan, Vipin Kumar. Data mining and knowledge discovery, 2004

350 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems

View Paper: https://www.semanticscholar.org/paper/e63d8dc37eefe3f5806db08e8d96492deefef285

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

Citations in the corpus

Outlier Analysis (2013, background): “and web sequence robot detection [440], that the nature of the anomalous sequences are highly specific and governed by the underlying application.”
Data Mining for Web Personalization (2007, methodology): “The latter task can be performed by maintaining a list of known spiders, using heuristics, or using classiﬁcation algorithms to build models of spider and Web robot navigations [135].”
Web Usage Mining as a Tool for Personalization: A Survey (2003, methodology): “Tan and Kumar (2002) propose a method for identifying spider sessions based on a variety of features that are extracted from access logs such as percentage of media ¢les requested, percentage of requests made by HTTP methods, as well as features that are showing the breadth-¢rst searching behavior…”
A brief survey on sequence classification (2010, background): “Other interesting examples include classifying query log sequences to distinguish web-robots from human users [58; 18] and classifying transaction sequence data in a bank for the purpose of combating money laundering [42].”
Adversarial Web Search (2011, background): “3 Fortunately, the crawlers that produce the referrer spam can often be detected as non-human [185, 215, 218], and thus their activities, in theory, can be filtered from usage logs.”
Understanding Website Behavior based on User Agent (2016, methodology): “Many techniques have been proposed to detect web crawlers, for example log analysis, heuristic-based learning techniques, traﬃc pattern analysis and human-detection tests [1, 5, 6, 7].”
A Framework for the Evaluation of Session Reconstruction Heuristics in Web-Usage Analysis (2003, background): “…user authentication, the activation of cookies that are installed at the user’s work area and can thus attach a unique identifier to her requests, as well as the replacement of a site’s static pages with dynamically generated pages that are uniquely associated with the browser that invokes them.”