What Yelp Fake Review Filter Might Be Doing?

By Arjun Mukherjee, Vivek Venkataraman, B. Liu, Natalie S. Glance. International Conference on Web and Social Media, 2013

668 citations recorded by the source corpus.

Online reviews have become a valuable resource for decision making. However, its usefulness brings forth a curse ‒ deceptive opinion spam. In recent years, fake review detection has attracted significant attention. However, most review sites still do not publicly filter fake reviews. Yelp is an exception which has been filtering reviews over the past few years. However, Yelp’s algorithm is trade secret. In this work, we attempt to find out what Yelp might be doing by analyzing its filtered reviews. The results will be useful to other review hosting sites in their filtering effort. There are two main approaches to filtering: supervised and unsupervised learning. In terms of features used, there are also roughly two types: linguistic features and behavioral features. In this work, we will take a supervised approach as we can make use of Yelp’s filtered reviews for training. Existing approaches based on supervised learning are all based on pseudo fake reviews rather than fake reviews filtered by a commercial Web site. Recently, supervised learning using linguistic n-gram features has been shown to perform extremely well (attaining around 90% accuracy) in detecting crowdsourced fake reviews generated using Amazon Mechanical Turk (AMT). We put these existing research methods to the test and evaluate performance on the real-life Yelp data. To our surprise, the behavioral features perform very well, but the linguistic features are not as effective. To investigate, a novel information theoretic analysis is proposed to uncover the precise psycholinguistic difference between AMT reviews and Yelp reviews (crowdsourced vs. commercial fake reviews). We find something quite interesting. This analysis and experimental results allow us to postulate that Yelp’s filtering is reasonable and its filtering algorithm seems to be correlated with abnormal spamming behaviors.

Subfields: Artificial Intelligence, Information Systems, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/6d3e5feee7f03606ae2fe11c165846abe5f360a3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Sociology and Political Science

Citations in the corpus

False Information on Web and Social Media: A Survey (2018, background): “[62] showed that fake reviews were shorter than real reviews, and Ott et al.”
Spotting opinion spammers using behavioral footprints (2013, background): “Although these reviews are fake, they do not reflect the dynamics of fake reviews in a commercial website [31] as the Turkers do not have the same psychological state of mind when they write fake reviews as that of fake reviewers in a commercial website who have real business interests to promote…”
Automated Crowdturfing Attacks and Defenses in Online Review Systems (2017, methodology): “We use the percentage of word count in each class as a feature, and exclude the features already included in the previous groups [48, 49].”
What Online Reviewer Behaviors Really Matter? Effects of Verbal and Nonverbal Behaviors on Detection of Fake Online Reviews (2016, methodology): “mind when writing fake reviews in comparison to actual fake review writers, causing the accuracy of detection models constructed based on those pseudo-fake reviews much higher than the accuracy of models constructed based on real-world fake reviews [46, 47].”
Detecting opinion spams and fake news using text classification (2018, background): “Mukherjee et al. 24 built a so-called author spamicity model (ASM) to detect suspicious reviewers based on their behavior footprint.”
Fake News: A Survey of Research, Detection Methods, and Opportunities (2018, methodology): “…types of information, e.g., online communications [Hancock et al. 2007; Pak and Zhou 2015; Rubin 2010; Zhou et al. 2004b], reviews [Li et al. 2014; Mukherjee et al. 2013b; Ott et al. 2011; Popoola 2018; Shojaee et al. 2013; Zhang et al. 2016], statements [Fuller et al. 2009; Humpherys et al.…”
A survey on sentiment analysis methods, applications, and challenges (2022, background): “Unstructured Sentiment is an informal and free-flowing writing type in which the writer is not constrained by any rules (Mukherjee et al. 2013).”
REV2: Fraudulent User Prediction in Rating Platforms (2018, background): “• ICWSM’13 [24] uses user’s fraction of positive reviews, maximum reviews in a day.”
Towards a General Rule for Identifying Deceptive Opinion Spam (2014, background): “Recently there has been increasing concern about deceptive opinion spam (Jindal and Liu, 2008; Ott et al., 2011; Wu et al., 2010; Mukherjee et al., 2013b; Wang et al., 2012).”
Collective Opinion Spam Detection: Bridging Review Networks and Metadata (2015, methodology): “…rating deviation of rating from j average rating: ⇤ ⇤ H Weighted rating deviation [15], where reviews are weighed by recency: , for H Burstiness [5, 20]—spammers are often short-term members of the site. where H Extremity of rating [18]: = 1 for ratings { 5 } , 0 otherwise (for { 2 , 3 } ) DEV H…”
Exploiting Burstiness in Reviews for Review Spammer Detection (2021, background): “Mukherjee et al. (2013) classified Yelp filtered and unfiltered reviews, and performed a comparative study of commercial vs. crowdsourced fake reviews for supervised classification.”
Algorithmic Accountability (2015, methodology): “I analyzed the cases presented above, together with the interviews of key informants and a review of algorithm reverse engineering in the literature (Baker and Potts 2013; Guha, Cheng, and Franci 2010; Lazer et al. 2014; Mikians et al. 2012; Mukherjee et al. 2013; Hannak et al. 2013; Sweeney 2013).”
Algorithmic Accountability Reporting: On the Investigation of Black Boxes (2014, background): “…cases presented in the last section, as well as other examples of reverse engineering algorithms published in academic or non-mainstream outlets, 33,34,35,36,37 there are several key challenges to launching an investigation into an algorithm: identifying a meaningful target, sampling the…”