fix(ops): promotion's frozen copy fills its temp folder entry by entry by Nathan-Mulligan · Pull Request #2145 · LaunchAstro/ops-astro · GitHub
Skip to content

fix(ops): promotion's frozen copy fills its temp folder entry by entry - #2145

Merged
Nathan-Mulligan merged 1 commit into
mainfrom
fix/2124-promotion-copy-node-24-22
Oct 20, 2026
Merged

Nathan-Mulligan merged 1 commit into
mainfrom
fix/2124-promotion-copy-node-24-22

Conversation

@Nathan-Mulligan

@Nathan-Mulligan Nathan-Mulligan commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Why: since Node 24.22.0 (nodejs/node#64124) cpSync with errorOnExist throws ERR_FS_CP_EEXIST on an existing destination folder, and frozenCopy copied into the folder mkdtemp had just made, so node scripts/ops/promote.mjs refused every promotion on 24.22 or later.
Left undone: #2144
Risk: sensitive (deploy and promotion), low: 8 product lines in one function. The copy is the same 0700 temporary folder and every other step (served modes, digest check, swap) is unchanged. The one behaviour cpSync gave for free, refusing a source that is a link, is now an explicit lstat check.
Closes #2124
Section: ci

What changed

  • scripts/ops/served-copy.ts:73-82: frozenCopy copies each of the artefact's entries to its own new name in the mkdtemp folder (cpSync(join(src, name), join(copy, name), { recursive: true, errorOnExist: true, force: false })), instead of copying the artefact folder onto the folder that already exists. errorOnExist and force: false stay, so anything already at a name is still refused. Product +8/-1, tests 0.
  • Before listing, the artefact must still be a real folder (lstatSync(...).isDirectory()), else it is refused as changed. cpSync's own lstat of the source refused a link put in the artefact's place; readdirSync would follow one (tests/ci/promotion-trust-chain.test.ts "an artefact swapped for a link after selection" holds this).

Why not the ticket's join(copy, 'bytes') shape

Built first (3792e6c, never pushed). It passed the three named files but failed three in tests/ci/promotion-copy-replace.test.ts on both Node versions. Those tests inject their faults on the copy at served/.copy-<x> and its rename from that path, and check that folder stays closed while its entries are set. With the copy one level down, they no longer test what they say. Copying entry by entry keeps the copy at that path, so every test stands unedited.

Proofs (Mac arm64, nodejs.org binaries checked against SHASUMS256)

Run Node 24.22.0 Node 24.21.0 Node 26.3.0
Red, base d91fc0e: frozen-copy + trust-chain + p12 12 failed / 27, all ERR_FS_CP_EEXIST 27 / 27
Green, head 8496438: frozen-copy + trust-chain + copy-replace + p12 35 / 35 35 / 35 35 / 35
tests/ci + p12, head 8496438 1234 passed, 2 failed, 319 skipped 1234 passed, 2 failed, 319 skipped

The tests/ci failures are not this change. Both Node versions: queue-width-and-the-end-of-batch-run.proof hit ERR_MODULE_NOT_FOUND in its .local/self-test copy during the parallel run, and is 4 / 4 alone on each version. On 24.22.0: production-unit-parks-and-runs-the-worker.proof. On 24.21.0: staging-containment S0-1. Both of those are Docker: all predefined address pools have been fully subnetted on this Mac's shared daemon. None of the three imports promotion code.

Checklist

  • Data separation, money, permissions, web writes: no query, record, screen or grant changed; no business, client or person data touched.
  • Secrets and sign-in: no credential path changed.
  • Races and timing: the artefact can still be swapped after selection. The lstat check refuses a link that is there when the copy starts. A swap after that check is copied from the link's target, then has to pass holds(copy, digest) on the promoter-only copy, which refuses any link inside and any bytes that differ. Before this change the window sat inside cpSync, between its lstat and its read. Now it sits between the lstat and readdirSync, with the same result.
  • Tests that prove their claim: red is main's own tests on 24.22.0, green on both versions at head; no test edited. Undoing the fix brings back the 24.22.0 red (the base row). Dropping the lstat check fails the trust-chain link test on every version (seen at an intermediate build).
  • Guard and scanner bypasses: readdirSync never returns . or .. and a name holds no /, so every target sits directly in the fresh folder.
  • Docs match the code: the frozenCopy doc comment still holds (same temporary folder, closed until its entries are set); no doc names the copy step.

Security review: run against 8496438, no findings
What it checked: link at the artefact's top level and as direct children (copied as links, then refused by the digest); the lstat-to-readdir swap (only promoter-owned real files holding the digest are accepted); names escaping the copy folder; the copy folder staying 0700 until served modes, including 24.22's #65488 mode change; a failed promotion leaving served/ and the link unchanged. Noted, not introduced: a store writer racing link swaps could get files into the copy, which servedModes opens to others before holds rejects it. Every subfolder was already exposed this way before; this change only adds the top level. The reviewer's suggested hardening is to hash before opening the copy.

Review checkpoint

review-preflight: ready.

Review checkpoint
  branch:      fix/2124-promotion-copy-node-24-22
  base:        origin/main
  merge base:  d91fc0ee06551c712ccaf82cfbd41e577e25848b
  head:        8496438f1cc940d5e7077ba7ab3c189cb1754453
  commits:     1
    8496438f1 fix(ops): promotion's frozen copy fills its temp folder entry by entry
  files:       1
    scripts/ops/served-copy.ts

  review command: git diff d91fc0ee06551c712ccaf82cfbd41e577e25848b...8496438f1cc940d5e7077ba7ab3c189cb1754453

  This head is what the review covers. Any commit after it invalidates
  the review and the checks: run both again and record the new head.

Review record

Reviewer: Sol (Codex)
Model: gpt-6.1-sol
Head SHA: 8496438
Verdict: approve
Code review: no findings

Sol's approval of 8496438 (round 2, rules PASS, open findings 0, open disputes 0; S2145-1 dispute upheld, .nvmrc un-pin is #2144): #2145 (comment)
Director read of 8496438: pass (deploy/promotion), #2145 (comment)

Since Node 24.22.0 (nodejs/node#64124) cpSync with errorOnExist throws
ERR_FS_CP_EEXIST when the destination folder exists. frozenCopy copied
the artefact into the folder mkdtempSync had just made, so every
promotion on 24.22 or later was refused before anything was migrated.
It now copies each of the artefact's entries to its own new name in that
folder, keeping errorOnExist and force: false, so anything already at a
name is still refused. The copy is the same private temporary folder as
before, closed to others until its entries take their served modes.
Listing the artefact by name would follow a link put in its place, which
cpSync's own lstat refused, so a source that is no longer a real folder
is refused as changed, as it was.

Closes #2124

Agent-model: anthropic/claude-opus-5-5
Agent-tool: Claude Code
Assisted-by: LLM
Signed-off-by: Nathan Mulligan <183608168+Nathan-Mulligan@users.noreply.github.com>
@Nathan-Mulligan Nathan-Mulligan added the sensitive Sensitive lane: security, privacy, money or a guard over untrusted input label Oct 19, 2026
@Nathan-Mulligan

Copy link
Copy Markdown
Contributor Author

Director read 8496438: pass

Scope: deploy/promotion integrity (the BLOCKED reason), plus sign-in, money and data separation, on d91fc0e..8496438 (scripts/ops/served-copy.ts, 9 lines). Sol covers the rest.

  • The frozen copy now fills its fresh mkdtemp folder entry by entry instead of copying onto it, so Node 24.22's cpSync no longer refuses (served-copy.ts:79-82). The same options (errorOnExist: true, force: false) still hold per entry.
  • A link swapped in for the artefact is still refused: lstatSync(...).isDirectory() comes before listing (:77-78), keeping what the single cpSync used to refuse.
  • Whatever is copied still has to match the selected digest before it's served (holds, :111-113, unchanged after the loop), so a change between the check and the listing is refused fail-closed, not promoted.
  • Sign-in, money and data separation: not touched. No .nvmrc or test edits.

Until this lands, Tuesday's promote steps keep running on Node 24.21.0 by explicit path (register 196).

@Nathan-Mulligan

Copy link
Copy Markdown
Contributor Author

Finding: S2145-1
Criterion: 1
.nvmrc:1 The file still reads 24.21.0, so CI remains pinned instead of following the Node 24 line as #2124 explicitly requires this PR to restore.
Proof: tests/ci/sol-pr2145.proof.test.ts 'Sol proof, criterion 1: CI follows the Node 24 line after the promotion repair'

Notes: The proof fails on this head on both required Node versions: expected 24, received 24.21.0; this is an unfulfilled numbered requirement, although the temporary pin is inherited from the base.

Notes: Independently ran the three required suites, unchanged, on the Mac host using Node 24.22.0 and 24.21.0. At this head each runtime passes all 27 tests. Temporarily restoring only scripts/ops/served-copy.ts from d91fc0e reproduces 12 failures / 15 passes on 24.22.0, including the specified ERR_FS_CP_EEXIST refusal; 24.21.0 passes 27/27. The base 24.22.0 run also reported an ENOTEMPTY teardown error; restored-head runs did not. Product code and index were restored to the exact reviewed head.

Notes: Two additional proofs pass on both runtimes: private source modes plus umask 077 preserve served directory, ordinary-file and executable-file modes and independence from later store writes; a planted destination entry is refused with ERR_FS_CP_EEXIST and its temporary copy is cleaned. Final combined runs on each runtime have 29 passes and only S2145-1's proof failing. No further correctness or always-blocks finding identified.

Separation: No business, client or person separation is touched by this diff.

Coverage: All changed code, its relied-on copy/digest/promotion/trust dependencies and the three required test files were read; all required local runs and reviewer proofs ran. No binary paths or changed test files occur in the range. Merge-group shard 3 remains enforced by the merge gate, not a separate local coverage hold.

Notes: Hosted checks on this exact head currently include failed review evidence for this revision, local checks, database conformance and isolation tests; contamination gate and JavaScript/TypeScript analysis remain in progress. These states are left to the merge gate.

Notes: Proof file is uncommitted at tests/ci/sol-pr2145.proof.test.ts in the assigned worktree. A copy and execution logs are retained at /Users/nathan/_orchestration/rounds/2026-10-18-ops-live-agent-run/polish/sol-pr2145-8496438-f7f50mz9/proofs/. The pre-existing untracked .metadata_never_index was preserved.

Open findings: 1 (S2145-1)
Open disputes: 0 (none)
Reviewer: Sol (Codex)
Model: gpt-6.1-sol
Head SHA: 8496438
Verdict: changes requested
Code review: changes requested

@Nathan-Mulligan

Copy link
Copy Markdown
Contributor Author

Dispute: S2145-1 (.nvmrc still reads 24.21.0)

The finding is accurate: #2124's done-when includes .nvmrc reading 24. This PR leaves that out on purpose, and nothing else in #2124 is left.

Evidence: the lane's brief (round 2026-10-18-ops-live-agent-run, briefs/OA-Fix-2124.md, Job step 2) says "Never edit a test to go green and never touch .nvmrc", so the lane may not make this change. The un-pin is its own ticket, #2144, listed under Left undone: in this PR's body. Its done-when is the same line plus the merge group's shard 3 green on the Node setup-node resolves. Keeping it separate also means one PR moves CI's Node for every merge group, and it can be reverted alone: this lane proved tests/ci on 24.22.0 (1234 pass; the 2 failures are a parallel self-test collision that passes alone and Docker address pools), not the full suite or the database suites.

Your other proofs (umask 077 with private source modes, a planted destination entry refused with ERR_FS_CP_EEXIST) agree with this head. No change pushed; head stays 8496438.

If the orchestrator rules that the brief's line only meant "do not fix this by pinning", the fix is the one line .nvmrc 24.21.0 to 24 in round 2, with #2144 closed by it.

@Nathan-Mulligan

Copy link
Copy Markdown
Contributor Author

@Nathan-Mulligan Nathan-Mulligan added the ci-next PR is next in line for Sol or the merge queue: run the full PR CI label Oct 19, 2026
@Nathan-Mulligan
Nathan-Mulligan added this pull request to the merge queue Oct 19, 2026
Merged via the queue into main with commit 02f318f Oct 20, 2026
55 of 80 checks passed
@Nathan-Mulligan
Nathan-Mulligan deleted the fix/2124-promotion-copy-node-24-22 branch October 28, 2026 02:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-next PR is next in line for Sol or the merge queue: run the full PR CI sensitive Sensitive lane: security, privacy, money or a guard over untrusted input

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Promotion's frozen copy is refused on Node 24.22 (cpSync errorOnExist into its own temp folder)

1 participant