Search code examples
javascriptperformancebenchmarkingjsperf

Which JS benchmark site is correct?

I created a benchmark on both jsperf.com and jsben.ch, however, they're giving substantially different results.

JSPerf: https://jsperf.com/join-vs-template-venryx
JSBench: http://jsben.ch/9DaxR

Note that the code blocks are exactly the same.

On jsperf, block 1 is "61% slower" than the fastest:

On jsbench, block 1 is only 32% slower than the fastest: ((99 - 75) / 75)

What gives? I would expect benchmark sites to give the same results, at least within a few percent.

As it stands, I'm unable to make a conclusion on which option is fastest because of the inconsistency.

EDIT

Extended list of benchmarks:

Not sure which is the best, but I'd skip jsben.ch (the last one) for the reasons Job mentions: it doesn't display the number of runs, the error margin, or the number of operations per second -- which is important for estimating absolute performance impact, and enabling stable comparison between benchmark sites and/or browsers and browser versions.

(At the moment http://jsbench.me is my favorite.)

Solution

  • March 2019 Update: results are inconsistent between Firefox and Chrome - perf.zone behave anomalously on Chrome, jsben.ch behaves anomalously on Firefox. Until we know exactly why, the best you can do is benchmark on multiple websites (but I'd still skip jsben.ch, the others give you a least some error margin and stats on how many runs were taken, and so on)

    TL;DR: running your code on perf.zone and on jsbench.github.io (see here and here), the results closely match jsperf. Personally, and for other reasons than just these results, I trust these three websites more than jsben.ch.

    Recently, I tried benchmarking the performance of string concatenation too, but in my case it's building one string out of 1000000+ single character strings (join('') wins for numbers this large and up, btw). On my machine the jsben.ch timed out instead of giving a result at all. Perhaps it works better on yours, but for me that's a big warning sign:

    http://jsben.ch/mYaJk

    http://jsbench.github.io/#26d1f3705b3340ace36cbad7b24055fb

    https://run.perf.zone/view/join-vs-concat-when-dealing-with-very-long-lists-of-single-character-strings-1512490506658

    (I can't be bothered to ever have to deal with jsperf's not all tests inserted again, sorry)

    At the moment I suspect but can't prove that perf.zone has slightly more reliable benchmark numbers:

    • when optimising lz-string I used jsbench.github.io for a very long time, but at some point I noticed there were impossibly large error margins for certain types of code, over 100%.

    • running benchmarks on mobile is fine with jsperf.com and perf.zone, but jsbench.github.io is kinda janky and the CSS breaks while running tests.

    Perhaps these two things are related: perhaps the method that jsbench.github.io uses to update the DOM introduces some kind of overhead that affects the benchmarks (they should meta-benchmark that...).

    Note: perf.zone is not without its flaws. It sometimes times out when trying to save a benchmark (the worst time to do so...) and you can only fork your own code, not edit it. But the output still seems to be more in line with jsperf, and it has a really nice "quick" mode for throwaway benchmarking