You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[CHORE] perf: cheaper unshare() copy for store-linked packages that run install scripts (#1918 follow-ups) #1934
#1918 hardlinks packages with install scripts from the global store like any other package and, right before one of their lifecycle scripts runs, replaces their shared files with private copies (src/graph/src/reify/unshare.ts), so the script cannot write into the store. On the default install, which runs no scripts, that is a clear win (large −5…−8.5 %, babylon −3…−6 %, see the A/B on the PR). On installs that opt into running a store-linked package's scripts it is a cost: the package is linked and then copied, where before #1918 it was copied once at placement. That path is install / ci with an --allow-scripts query that matches the package, and vlt build on a tree installed by #1918 (once per tree).
Measured (same A/B, large: core-js@3.50.0, core-js@2.6.12, es5-ext@0.10.64, 5,647 files; install with VLT_ALLOW_SCRIPTS=':scripts', warm store + lockfile, 20 runs per arm, 4 vCPU Linux box): +230 ms wall per install (+13.8 %, 1671 → 1902 ms median) and +830 ms CPU in both rounds (hyperfine user 1.19 / 1.24 → 1.47 / 1.49 s, system 0.76 / 0.75 → 1.32 / 1.33 s, round 1 / round 2) against the parent commit. The PR description's own figure is +158 ms (+7.5 %) on ci --allow-scripts=:scripts.
Why it costs more than the old copy. For every file with nlink > 1, unshare() does readFile + writeFile to a tmp sibling (wx, plus chmod for executables) + rename over the link, 16 files at a time, every step a hop through libuv's thread pool (4 threads by default, which vlt does not raise), on top of the link itself. The pre-#1918 placement copy in src/tar/src/link-tree.ts was one synchronous readFileSync + writeFileSync (wx, plus statSync + chmodSync for executables) per file on the main thread, with no link, no per-file rename and no thread-pool hops. On this box the system time of the install rises ≈75 %.
💡 Proposed follow-ups
Both are non-blocking for #1918 and independent of each other.
Cheaper copy in unshare(). Copy with fs.copyFile(p, tmp, COPYFILE_EXCL) (libuv copies in the kernel where it can, no buffer round trip through JS, mode preserved) and keep the rename(tmp, p); the safety property is unchanged, since the copy targets the new tmp inode and the shared inode is never written. Also try a lower concurrency than 16 (the default pool has 4 threads) or a plain synchronous loop like the placement copy, and measure on a small and a large box.
Copy at placement when the scripts will run in the same install. When the allow-scripts query matches a store-linked package, copying it at placement, as before graph: hardlink install-script packages, copy them only before their scripts run #1918, avoids the link-then-unshare round trip. Today the query is evaluated in reify_() after addNodes() / extract (filterNodesByQuery runs after checkNeededBuild(), just before binChmodAll() and build()), and it cannot simply be moved earlier: :scripts matches on node.manifest, which a lockfile install without an existing node_modules only has once the package is extracted, and on buildState, which checkNeededBuild() sets after extraction. Deciding copy-vs-link at placement therefore needs the allow-scripts result for the node before it is placed (e.g. from the store index's scripts flag plus the query): a design change rather than a tweak, and the first item is worth doing regardless.
Activity