@@ -10,6 +10,7 @@ directory, see [the guide on benchmarks](../doc/contributing/writing-and-running
1010## Table of Contents
1111
1212* [ File tree structure] ( #file-tree-structure )
13+ * [ ` node:bench ` evaluation tools] ( #nodebench-evaluation-tools )
1314* [ Common API] ( #common-api )
1415
1516## File tree structure
@@ -44,16 +45,92 @@ directories.
4445* ` common.js ` : see [ Common API] ( #common-api ) .
4546* ` compare.js ` : command line tool for comparing performance between different
4647 Node.js binaries.
48+ * ` compare-node-bench.js ` : parallel comparison tool for explicit ` node:bench `
49+ files. It does not change or invoke ` compare.js ` .
4750* ` compare.R ` : R script for statistically analyzing the output of
4851 ` compare.js `
4952* ` run.js ` : command line tool for running individual benchmark suite(s).
5053* ` scatter.js ` : command line tool for comparing the performance
5154 between different parameters in benchmark configurations,
5255 for example to analyze the time complexity. Pass ` --analyze ` to
5356 summarize the results without R.
57+ * ` scatter-node-bench.js ` : parallel scatter-data tool for an explicit
58+ ` node:bench ` file. It does not change or invoke ` scatter.js ` .
5459* ` scatter.R ` : R script for visualizing the output of ` scatter.js ` with
5560 scatter plots.
5661
62+ ## ` node:bench ` evaluation tools
63+
64+ The ` compare-node-bench.js ` and ` scatter-node-bench.js ` tools run explicit
65+ ` node:bench ` files without changing the existing benchmark framework or its
66+ tools. Each repeated observation for a benchmark identity is collected by a
67+ separate process invocation with one measured sample. Benchmarks declared in
68+ the same file still execute serially in that process and can share JIT, garbage
69+ collector, heap, and cache state. This differs from legacy configuration-level
70+ process isolation and must be considered when comparing the frameworks.
71+
72+ Compare two binaries and analyze the compatible CSV using ` compare.R ` :
73+
74+ ``` console
75+ ./node benchmark/compare-node-bench.js \
76+ --old ./node-main --new ./node-pr --runs 30 -- \
77+ benchmark/crypto/_create-hash.node-bench.js > compare-node-bench.csv
78+ Rscript benchmark/compare.R < compare-node-bench.csv
79+ ```
80+
81+ Pass ` --analyze ` to run the same Welch analysis inline. ` --max-regression N `
82+ implies ` --analyze ` and makes the command fail only when the Holm-Bonferroni
83+ adjusted p-value is below 0.05 and the full 95% confidence interval is worse
84+ than ` -N% ` . Requiring both conditions prevents a noisy point estimate from
85+ failing a regression gate.
86+
87+ ``` console
88+ ./node benchmark/compare-node-bench.js \
89+ --old ./node-main --new ./node-pr --runs 30 \
90+ --max-regression 5 -- benchmark/crypto/_create-hash.node-bench.js
91+ ```
92+
93+ Collect parameter data for the parallel buffer benchmark and plot it using
94+ ` scatter.R ` :
95+
96+ ``` console
97+ ./node benchmark/scatter-node-bench.js --node ./node --runs 30 -- \
98+ benchmark/buffers/_buffer-compare-offset.node-bench.js \
99+ > scatter-node-bench.csv
100+ Rscript benchmark/scatter.R --xaxis size --category method \
101+ --plot scatter-node-bench.png < scatter-node-bench.csv
102+ ```
103+
104+ Pass ` --analyze ` with an x-axis parameter to summarize the samples without R.
105+ The output includes mean and median confidence intervals, skew warnings, an
106+ optional bar chart, and Mann-Whitney U and Cliff's delta comparisons between
107+ consecutive x-axis values. Use ` --category ` for a second grouping parameter and
108+ ` --no-chart ` to omit the chart.
109+
110+ Because configurations in one file share a process, inline analysis averages
111+ aggregated configurations into one value per outer process. Consecutive
112+ x-axis comparisons use alternating, disjoint process sets so the unpaired
113+ Mann-Whitney test does not treat correlated values as independent samples.
114+
115+ ``` console
116+ ./node benchmark/scatter-node-bench.js --runs 30 --analyze \
117+ --xaxis size --category method -- \
118+ benchmark/buffers/_buffer-compare-offset.node-bench.js
119+ ```
120+
121+ A file passed to ` scatter-node-bench.js ` must use one logical benchmark name.
122+ Parameter values distinguish its configurations. The tool rejects unstable
123+ identities and names or parameters that would merge unrelated CSV groups.
124+
125+ The underscore-prefixed benchmark files are parallel ports used to compare the
126+ measurement frameworks. Legacy discovery ignores them, so the original files
127+ remain the source benchmarks for ` run.js ` , ` compare.js ` , and ` scatter.js ` . The
128+ ports use the platform-specific original relative filename as their benchmark
129+ name and preserve parameter column names to keep CSV grouping compatible. For
130+ a direct framework comparison, collect the same number of runs from an
131+ original benchmark with ` scatter.js ` and from its port with
132+ ` scatter-node-bench.js ` , then compare their rate distributions.
133+
57134## Common API
58135
59136The common.js module is used by benchmarks for consistency across repeated
0 commit comments