Measuring personalization of web search

By Anikó Hannák, Piotr Sapiezynski, Arash Molavi Kakhki, Balachander Krishnamurthy, David M. J. Lazer, Alan Mislove, Christo Wilson. 2013

451 citations recorded by the source corpus.

Web search is an integral part of our daily lives. Recently, there has been a trend of personalization in Web search, where different users receive different results for the same search query. The increasing personalization is leading to concerns about Filter Bubble effects, where certain users are simply unable to access information that the search engines' algorithm decides is irrelevant. Despite these concerns, there has been little quantification of the extent of personalization in Web search today, or the user attributes that cause it. In light of this situation, we make three contributions. First, we develop a methodology for measuring personalization in Web search results. While conceptually simple, there are numerous details that our methodology must handle in order to accurately attribute differences in search results to personalization. Second, we apply our methodology to 200 users on Google Web Search; we find that, on average, 11.7% of results show differences due to personalization, but that this varies widely by search query and by result ranking. Third, we investigate the causes of personalization on Google Web Search. Surprisingly, we only find measurable personalization as a result of searching with a logged in account and the IP address of the searching user. Our results are a first step towards understanding the extent and effects of personalization on Web search engines today.

Subfields: Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

Citations in the corpus

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, background): “Hannak et al. (2013) observe that, on average, about 12% of Google search results exhibit diﬀerences due to personalization.”
Filter Bubbles, Echo Chambers, and Online News Consumption (2016, unknown): “Additionally, search engines, news aggregators and social networks are increasingly personalizing content through machine learned algorithms (Agichtein et al., 2006; Das et al., 2007; Hannak et al., 2013), which can in principle create “filter bubbles” (Pariser, 2011) that amplify ideological segreg”
Measuring Price Discrimination and Steering on E-commerce Web Sites (2014, methodology): “This has motivated researchers to begin measuring the personalization present in deployed systems, such as web search engines [17, 27, 50] and recommender systems [4].”
Algorithmic Accountability Reporting: On the Investigation of Black Boxes (2014, background): “31 One of the issues is that sites like Staples might be using A/B testing to analyze whether or not subtle differences on their websites are useful to them.”
Auditing Search Engines for Differential Satisfaction Across Demographics (2017, background): “Other related studies measured the impact of demographics on search results [23], examined the search engine manipulation eﬀect [15], explored demographic context as a means to improve search results for ambiguous queries, and analyzed gender diﬀerences in search perceptions [45].”
Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media (2017, background): “Therefore, building on previous studies that have adopted this “black box” view for an algorithmic system while auditing it [4, 9, 14, 15, 20], we characterized the bias of the ranking algorithm in Twitter’s search platform, without knowing its internals.”
Algorithmic Accountability (2015, methodology): “I analyzed the cases presented above, together with the interviews of key informants and a review of algorithm reverse engineering in the literature (Baker and Potts 2013; Guha, Cheng, and Franci 2010; Lazer et al. 2014; Mikians et al. 2012; Mukherjee et al. 2013; Hannak et al. 2013; Sweeney 2013).”
FairTest: Discovering Unwarranted Associations in Data-Driven Applications (2015, background): “Our work also relates to the field of web transparency [7], [25]–[32].”
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems (2018, background): “Researchers generate new data and user accounts with the help of scripts, bots [44], and crowdsourcing [73] to emulate real data and real users in the auditing process.”
Discovering Unwarranted Associations in Data-Driven Applications with the FairTest Testing Toolkit (2015, background): “3), the field of web transparency [1,3,6,17,18,26,27,29,47,48] is closest to our work; indeed, our team contributed two scalable and generic system designs [26, 27].”
Peeking Beneath the Hood of Uber (2015, methodology): “[14] determined the features used by Google Search to personalize content, while [13,20,21] measure price discrimination and steering on e-commerce sites.”
Auditing radicalization pathways on YouTube (2019, background): “Previous large scale algorithmic auditing include measuring discrimination on AirBnB [14], personalization onweb search [19] and price discrimination on e-commerce web sites [20].”
Bias in Online Freelance Marketplaces: Evidence from TaskRabbit and Fiverr (2017, background): “Successful prior audits have looked at personalization on search engines [30, 35], localization of online maps [54], social network news-feeds [25], online price discrimination [31, 43, 44], dynamic pricing in e-commerce [15], and the targeting of online advertisements [29, 38].”