When searching online for a new gadget to buy or a movie to rent, many people pay close attention to the number of stars awarded by customer-reviewers on popular websites. But new research confirms what some may already suspect: those ratings can easily be swayed by a small group of highly active users.
Vassilis Kostakos, an assistant professor at the University of Madeira in Portugal and an adjunct assistant professor at Carnegie Mellon University (CMU), says that rating systems can tap into the "wisdom of the crowd" to offer useful insights, but they can also paint a distorted picture of a product if a small number of users do most of the voting. "It turns out people have very different voting patterns," he says, varying both among individuals and among communities of users.The article is not that great in my opinion. They want you to think that there is a problem with only a few people voting when the research is really discussing the fact that a small fraction of the user base accounts for the majority of total votes at places like IMDb. Movies made recently like Up and Inglorious Basterds rate respectfully at #39 and #43 while movies like Braveheart and Yojimbo rate in at #100 and #129. The problem is 18 year old kids with limited knowledge of cinema history rating a movie as a ten the same night they see it. While people over 40 are less likely to weigh in on a movie they saw years ago. For arguments' sake, lets say IMDb has a ratio of 75/20 That is 75 percent of all votes are cast by 20 percent of voters. Is this a problem? Does it invalidate the voting system?
The original paper which was presented at the 2009 IEEE International Conference on Social Computing and is posted on Kostakos's website here[PDF] is not about the unreliablity of crowd wisdom or collective intelligence at all. It examined three different voting systems and the accessibility and barriers to cast a vote. It did not even contain the word "distort" as suggested by the article. It mentions that the results are unreliable if the item only has a couple of votes and it does not seem to say anything on reliablity with respects to large numbers of votes or reviews. It considers the voting and rating system of Amazon.com but does not include the recomendation system, you know - that ability to rate other users comments, nor does it consider the feature by which it recommends other products people have bought or looked at.
So in my opinion, the research is just OK and the article about it sucks.
Posts
Research is business of collecting facts, and facts must be impartial. Articles take research and use it to prove a point of view, often regardless of the actual meaning of the data returned by the research.
ReplyDeleteThe research is interesting, and I'm certain that the research does not accurately portray the "average person"'s opinion on that particular movie, book, or what have you...but the "average person" is not the matter here. The statistics collected by the voting mechanism should be relevant to 1)the people that the product is relevant to and 2)to the people on that site, viewing that item
Thus the group that is in the majority in voting or commenting does swaying statistics, but this is a positive affect a this very same group is often the group that is in the majority in using the information generated by such voting systems.
IE: most internet statistics on the most used operating systems, internet browsers and internet connection types caution that they are based on what is probably a small, quite probably skewed section of the population. But the people most likely to be using gaming software, browsing websites, or requiring high connectivity are those in that skewed percentage.
I recently built myself a new computer and I definately paid attention to the number of stars and the reviews. It's true that an item with a small number of votes can't be trusted because you don't know if they are actual users or just fan boys of the company, intel or AMD fan boys for example. However in my experience most places have generally helpful reviews and accurate star ratings. Also if you think the site you are looking at has skewed ratings you can always find independent reviews to get the info you need to make a purchase decision.
ReplyDelete