I have a marketplace that is similar to how Fiverr works. Service providers say that a given "gig" or service will take a given number of days. I want to start giving sorting/visibility preference to the service providers that deliver on time or before for more weight. Those that fail to deliver in time will be penalized.
What are some ways that I can assign and track a weighted rating based on delivery performance over time? Would the following pseudo code be on the right track?
// Everyone has a starting weight of 1
on-time delivery = current weight * 1.1
early delivery = current weight * 1.2
late delivery = current weight * .9
fail to deliver = current weight * .75
Sort sellers by weight descending
I'm not sure these if these values make sense or would weight too quickly. Any help is appreciated!
Your approach is alright, but I would recommend trying exponential moving average.
Firstly, it takes into account previous samples (sample in this case would be the quality of delivery) with weight of alpha and the new sample with 1 - alpha. This means, previous samples have bigger impact than the just taken sample, which could be more fair to people who for example delivered always on time but suddenly have few bad deliveries. Their overall weight wouldn't drop that much as for someone who had less on time deliveries and suddenly has bad ones. This brings some kind of reputation element.
Second advantage is that the weights fluctuate less. There is a smaller variance. In your approach people with a lot of bad deliveries will have weights close to zero pretty quicky, and this could potentially lead to some numeric problems. And also people with a lot of good deliveries could have their weights too big, causing potential overflows.
I would recommend to choose alpha as 0.9 to 0.99. This value is usually used, for example in some neural network optimizers, like Adam, but of course you should try different values and see how the weights change. You can use the weight change factors as you proposed, it's fine.
Example update with alpha = 0.99:
weight = weight * 0.99 + 0.01 * 1.2 // early delivery
The smaller the alpha, the lesser the fluctuations and the more predominant is the history of samples over one sample.