Understanding Website Behavior based on User Agent
By Kien Pham, Aécio Santos, Juliana Freire. 2016
17 citations recorded by the source corpus.
Web sites have adopted a variety of adversarial techniques to prevent web crawlers from retrieving their content. While it is possible to simulate users behavior using a browser to crawl such sites, this approach is not scalable. Therefore, understanding existing adversarial techniques is important to design crawling strategies that can adapt to retrieve the content as efficiently as possible. Ideally, a web crawler should detect the nature of the adversarial policies and select the most cost-effective means to defeat them. In this paper, we discuss the results of a large-scale study of web site behavior based on their responses to different user-agents. We issued over 9 million HTTP GET requests to 1.3 million unique web sites from DMOZ using six different user-agents and the TOR network as an anonymous proxy. We observed that web sites do change their responses depending on user-agents and IP addresses. This suggests that probing sites for these features can be an effective means to detect adversarial techniques.
Subfields: Information Systems, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
- Fernando Diaz
- Alexandra Olteanu
- Carlos Castillo
- Emre Kıcıman
- Hamed Zamani
- Jimmy J. Lin
- W. Bruce Croft
- Nick Craswell
In Signal Processing
Citations in the corpus
- Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, background): “The latter may include detection methods that block access to clients suspected of being automatic data collection agents, resulting in an escalation of covert (“stealth”) crawling methods (Pham et al., 2016).”