How many TripAdvisor reviews are fake? And does it matter?

Full disclosure: I am not the first to write about this groundbreaking research in this blog post. The Economist beat me to it in their article  “TripAdvisor’s fake battle”. This article by Mayzlin, Dover & Chevalier and published on the Social Science Research Network in 2012 is called “Promotional Reviews: An Empirical Investigation of Online Review Manipulation”. As the title suggests, the focus of the paper is are customer reviews, specifically fake ones. They use an economist’s perspective to theorise who may profit the most from false positive and false negative reviews on travel sites. Then they test the theory using Expedia.com as a control group (as only those who book through Expedia can post reviews there) versus TripAdvisor as an experimental group (where anyone can post a review without evidence of ever having stayed at the accommodation). They do a whole bunch of other stuff to ensure their data is valid for comparison.

Before going to the take aways, there are two things to note. First, more research on inauthentic reviews have to do with what people say in their review. These authors pioneer a new way of looking at it, by looking at probabilities of reviews being fake by both the context of the review, reviewer and the object of the review. Second, this is the first study that uses a control and experimental group for reviews in this particular way. Third, these authors theorise about who is most likely to post fake reviews and then look at those specific profiles.

The Take Aways
This article really has to be read, because the reason the authors believe what they do, and then the way they go about testing their theory is really fantastic. I don’t like to gush, but this is Freakonomics territory. Some takeaways:

(1) The authors posit that accommodation providers most likely to post false reviews (positive on their site, negative on their adjacent competition’s site) are independently-owned or small operators (or both). Their study supports this.

(2) The authors posit that fake reviews are more likely to be either 1, 2 or 5 star. Fake reviews are more likely to come from single posters (reviewers with only one or two posts). The data bears out that this is likely to be the case.

(3) The authors posit there is more positive manipulation (fake good feedback about your own hotel) than negative manipulation (fake negative feedback about the competition in the neighborhood). The study bears this out.

(4) The authors propose that about five percent of reviews are fake for isolated small hotels. The authors propose if a hotel is not isolated, the number of fake reviews would climb to about 10 percent.

(5) Although there is more manipulation for positive reviews, the effect of a negative review on the average star rating of a hotel is more significant. Therefore, although there are less fake negative reviews, they carry a greater weight overall in affecting the hotels overall score.

(6) In their study, 23% of the TripAdvisor reviews would have been eligible to be considered fake. They were posted by one time reviewers and were on the extreme end of the spectrum.

The authors conclude that even though the proportion of reviews that can be considered fake is high, they are not attempting to prove that any one particular review is fake, and they do not believe that the proportion necessarily negatively affects the customer impressions on an open-ended system: “Our empirical results show that the hotels are essentially able to self-police so that while they engage in some manipulation, the amount is not big enough to overwhelm the informational value of the site.”

Objective Authenticity
This post focuses on Objective Authenticity. According to Wang (1999), Objective Authenticity is when something is what it claims to be. Whereas one of the sites studies uses a system where only those who book through the system can leave a review, the other is an open system, meaning anyone can post. All you need is an email.

Objective authenticity is really important for these site’s credibility. An open system is attractive because it casts the widest net for opinions; however it leaves the largest margin for duplicity.

Moreover, an averaging star rating based on reviews means that negative reviews (fake or authentic) weigh more significantly than positive ones. So those with an intent to punish n establishment, are more incentivised to generate a lower score.

TripAdvisor has mechanisms to attempt to weed out fakes. They have an internal system, as well as the ability for hotel operators to identify reviews that may by fake. However whether something is fake or not is left to TripAdvisor to adjudicate.

Sure, there is a bit of caveat emptor here. If one is using a site to make a buying decision, it would behove one to understand how the site works. However, presumably the credibility of the site, and the reviews on it, also something that grows and changes as the person uses, or doesn’t use the site.

However, if users of the site found that the reviews on the site did not mirror their experience in some way, it would not be as successful as it is. Existential Authenticity, is where someone’s experience is consistent with a product expectation, with enough variance to engender genuineness, develops when one’s experience is close enough to TripAdvisor for the user to decide it is a useful guide, without the expectation that their experience is exactly what TripAdvisor says.

Advertisements