Building a ratings system

Source: https://rateyourmusic.com/list/jweber14/my_rating_system/

Ratings are ubiquitous in consumer internet space. Driver & riders on Uber, service providers on marketplaces, doctors on Practo, products and sellers on Amazon, hotels on Tripadvisor, restaurants on Yelp, all have ratings. Ratings are an essential and critical component of decision making. Each business has its own requirement and engineers a rating system that suits its need (which may change depending on stage and scale)

This article lays out the objectives and various components of a rating system.

Objectives of a rating system:

1. True reflection of user experience
2. Enough spread to allow for easier decision making
3. Filters spam— no brainer but tough to crack
4. Hard to game/ cheat — an ever going battle

Here are some essential things to keep in mind while engineering a ratings system:

Type of ratings/reviews

Ratings can have detailed reviews associated with them or could be just plain simple ratings. Reviews on a platform may have different attributes such as length of text, presence of photos, number of photos associated, social sharing enabled, number of fields filled etc. Depending upon the type of rating and the type of review associated with the rating, you can devise a system to give different weights to different attributes.

User Credibility

Not all users are equal when it comes to content contribution. One can easily segment the users in high/mid/low contributing users. The definition of these segments may include parameters like contribution over a fixed length of time, kind of content (ratings, reviews, photos, error-reports etc), social connections etc. You need to create a scale to grade users based on their level of contribution and their usefulness (likes, up-votes etc) to rest of the consumers.

Users who contribute high quality and quantity should get higher weights in the overall ratings calculation that those who have 1 or 2 contributions.

Time Recency

Businesses are always in a state of flux and a lot of things change every few months. A restaurant may start offering new cuisine, service providers may not be as punctual as they were 4 months ago etc. Thus one needs to give differential weights to ratings and reviews written today versus those that were written 6 months ago. Here’s a sample time decay curve that you can experiment with.

Cold Start Problem

How do you handle new entrants — hotels, restaurants, drivers, service providers etc? This is a chicken and an egg problem. If you don’t have enough rated places, users will not find the platform particularly useful for decision making. And if you don’t have enough returning users, you will not be able to make a meaningful ratings system. This is the cold start problem and there are multiple ways of solving this:

1. Average base rating to all new entrants: All new entrants start at an average rating and as user votes trickle they move up or down the ladder.
2. Expert ratings/reviews: You can give experts/critics early access to the platform and seed some ratings for new establishments (Critical mass is important here)
3. New on platform tag: You can give a new on platform tag to new entrants so that users give them a chance and improve rating velocity of the place.

Normalization

This one is tricky. User generated content (UGC) velocity is not very high. Typically around 2–4% of your users will contribute content and most of them will not be sticky. You may end up in a situation where a lot of places/providers will be around the same rating zone. This situation doesn’t really aid decision making as users are not be able to decipher anything meaningful if all places are in the same rating range.

This is where normalization will help. But you have to be really careful as simple normalization may result in some players getting undue advantage and some of them getting penalized disproportionately. You can use multiple slopes, multiple cutoffs, thresholds etc. to address this problem of extremes.

In conclusion, there is no one size fits all approach that one can take to develop a ratings system. You need to build one that is relevant to the business and the current context. You will need to relook it every few months.