Repository navigation
State of Benchmark #1
Description
Hey, I liked your article but I think you could include more information to make it a reference when we talk about available benchmark tools on Node.
I don't know if you didn't add more benchmarks because you don't know them or if it was intentional but I was doing the research to include the benchmark module on Node and I used this repository to compare the benchmarks: https://github.com/H4ad/benchmarks-comparisons.
You can run run.sh and you will have a lot of log files for each benchmark multiple times, which you can use to compare different tools. Some of them are more accurate and some of them are too noisy for some benchmarks (eg: tinybench). I also added inside that issue of introducing benchmark module on node a table comparing the different features, maybe could be interesting for you.
I'm also working on another benchmark with @RafaelGSS that was the initial proposal to include on Node, you can find it at https://github.com/RafaelGSS/bench-node (currently not available on npm).
This new benchmark may have a memory heap report like isitfast, with an output like:
create empty circuit x 242,837 ops/sec +/- 0.12% (8 runs sampled) heap usage=3.05Kb (3.02Kb ... 3.09Kb) min..max=(3.65us ... 5.01us) p75=4.41us p99=5.01us
Let me know if you have any questions about one of these benchmarks, or feel free to close this issue if you think this is unnecessary.
Activity
webpro commentedon Dec 23, 2023
Thanks, @H4ad!
Here's what I did (not) find when writing the article:
Only now I see your comprehensive table - no idea how I missed that. It's super interesting! Helps a lot.
So, based on your reply and the table:
@H4ad WDYT? Based on your feedback, when I have some more time (after Christmas), I plan to update the article.
Ethan-Arrowood commentedon Dec 23, 2023
I'm also away for the holidays, but jsperf.dev is definitely stagnated. I simply ran out of motivation to keep working on it when I realized there was little to gain from generalized benchmarking. I'm a big fan of open telemetry now and if I was to work on jsperf again I would most likely have it integrate directly with OT.
Let me know if there's any other context about the project. Happy holidays ❄️
webpro commentedon Dec 23, 2023
Thanks for the quick reply, Ethan. Wish you all the best 🎄☃️
H4ad commentedon Dec 24, 2023
About maintainability, I think all of them (except tinybench) lack a little bit about that, but I think those libraries didn't take too much maintenance effort since once you finish it, you will have too much to change.
About
isitfast, the author has plans to start to work again on it in the next year (the new API/version looks interesting too), and I got the idea of benchmarking the memory from this library, so at least give a mention about their innovation (besides is not production ready).Yes, our idea is to attract more ideas and feedback about it, try to make that benchmark battle-tested, and then in the future use that lib as the foundation for the
node:benchmarkmodule.About
jsperf.dev, it was suggested to me but I forgot about that during the research, so that's why I didn't include it. Their implementation looks a little bit similar to the new implementation ofisitfast, which I think was too exotic compared to the other benchmarks, although I think that implementation helps with isolation and probably makes it very easy to introduceworkerslikecronometro.About that feature comparison table, let me know if you don't understand the terms I use.
webpro commentedon Dec 24, 2023
Thanks a lot for all the feedback and info, @H4ad. Let me know if you like the updated article better, but happy to hear any more feedback you may have!
Update micro benchmarking tools (resolves #1)