Repository navigation
The symlink-escape guard now fails when it stops working - #172
Conversation
| * would pass for the wrong reason and would keep passing with the guard removed. | ||
| */ | ||
| function tree(files: Record<string, string>): string { | ||
| const dir = realpathSync(mkdtempSync(join(tmpdir(), 'bt-safety-'))); |
There was a problem hiding this comment.
REVIEWER: [medium] I ran this mutant and it survives: change const rootReal = realpathSync(root) to const rootReal = resolve(root) in safety.ts and the file still reports 22 pass / 0 fail. That is the one line that makes an unresolved caller path work, and nothing here pins it.
The reason it is unreachable is this realpathSync on the fixture root, and the justification given for it (repeated in the why-block Decisions) is not what the code does. I probed it on this machine:
tmpdir() = /var/folders/3y/.../T
raw fixture root = /var/folders/3y/.../T/bt-probe-Ehbt5V
realpath root = /private/var/folders/3y/.../T/bt-probe-Ehbt5V
root is a symlinked path? true
UNRESOLVED root + in-tree symlink => ACCEPTED (no throw)
enforceSourceBudget resolves the root itself (realpathSync(root)) before comparing, so an unresolved /var/... root does not make in-tree symlinks look like escapes. The claim "without this every in-tree symlink looks like an escape, so the tests would pass with the guard deleted" is false — the accept test would have failed, not passed.
What realpathSync here is actually load-bearing for is the harness's own string assertions (line 297 expect(realpathSync(link)).toBe(outside), line 298 outside.startsWith(root + '/'), and the ${root}-evil sibling). That is a fine reason to keep it — but it normalises away the exact production shape the guard's own realpathSync exists for, and the comment sells the opposite.
Suggested improvement: keep the resolved root for the existing fixtures, but add one test that passes the unresolved path (mkdtempSync(...) without realpathSync) containing an in-tree symlink and asserts .not.toThrow(). That test kills the resolve(root) mutant. Then rewrite this docblock (and the Decisions bullet) to say what is true: the resolved root is needed by the harness's path comparisons, not by the guard.
| // stepped over, and the only observable consequence is that MORE is counted. So the budget is | ||
| // the instrument: a cap that fits the source files alone must survive the presence of a large | ||
| // skipped directory, and must not survive that directory being walked. | ||
| for (const skipped of ['.git', 'node_modules', 'dist', 'build', '.next', 'coverage']) { |
There was a problem hiding this comment.
REVIEWER: [medium] These six tests, plus the negative control below, do not constrain where the skip check sits relative to the symlink branch — and that ordering is the traversal guard. I ran the mutant: hoist if (SKIP_DIRS.has(name)) continue; to the top of the for loop, above lstatSync(p) (a plausible "don't lstat what we skip anyway" optimisation), and the file still reports 22 pass / 0 fail.
Under that mutant a hostile repo commits ln -s /etc node_modules (or .git, dist, build, .next, coverage — six attacker-choosable names, all conventionally gitignored so nobody looks) and the escape check never runs on it. For a suite whose stated premise is "removing the guarantee turns something red", this is the same class of hole the PR was opened to close.
Suggested improvement: add one test — a symlink named a skipped directory pointing outside the root — asserting toThrow(/symlink escapes clone root/). One case (node_modules → outsideTree()) is enough to pin the ordering; a loop over all six is cheap if you want it.
| // A budget nobody pins can be "tuned" to infinity in a one-line diff. These three numbers | ||
| // are the whole guard when no caller passes an override, and `enforceSourceBudget`'s | ||
| // default parameter is what the container actually uses. | ||
| expect(DEFAULT_BUDGET).toEqual({ |
There was a problem hiding this comment.
REVIEWER: [medium] This test's own comment says the point is that "a budget nobody pins can be 'tuned' to infinity in a one-line diff" and that "enforceSourceBudget's default parameter is what the container actually uses" — but it only asserts the shape of the exported constant. No test ever calls enforceSourceBudget(root) with the budget omitted, so nothing connects the constant to the default parameter.
I ran the mutant: change the signature to budget: SafetyBudget = { maxFileBytes: Infinity, maxTotalBytes: Infinity, maxFiles: Infinity }, leaving DEFAULT_BUDGET exported and untouched — 22 pass / 0 fail. That is precisely the one-line tune-to-infinity the test claims to prevent, and it is the path that matters most: enforceSourceBudget has no in-repo caller (only index.ts re-exports it), so the container almost certainly calls it with no second argument.
Suggested improvement: add a test that exercises the default parameter — e.g. build a tree with one file over DEFAULT_BUDGET.maxFileBytes and assert enforceSourceBudget(root) (no budget) throws /file exceeds 8388608 bytes/. A cheaper alternative that also kills the mutant is expect(enforceSourceBudget.length).toBe(1) plus a no-arg call on a fixture built from DEFAULT_BUDGET's own numbers, but the real-tree version is the honest one.
| // PROMISE 1 + 2 — SYMLINKS ARE NOT FOLLOWED, AND ONE THAT ESCAPES IS REJECTED. | ||
|
|
||
| describe('the path-traversal guard', () => { | ||
| test('rejects a symlink whose target resolves outside the clone root', () => { |
There was a problem hiding this comment.
REVIEWER: [low] The guard's condition has two clauses — target !== rootReal && !target.startsWith(rootReal + sep) — and this block only pins the second. I ran the mutant: delete target !== rootReal && and the file still reports 22 pass / 0 fail.
The surviving behaviour is a false rejection, not a bypass, so it is not a security hole — but a repo that commits ln -s . self (or any link resolving to the clone root itself) would then fail its whole ingest with "symlink escapes clone root", and nothing here would catch the regression. The -evil sibling test at line 152 covers the separator; this clause has no counterpart.
Suggested improvement: add a test that a symlink pointing at the clone root itself is accepted — symlinkSync(root, join(root, 'self')), expect(...).not.toThrow().
| import { enforceSourceBudget, DEFAULT_BUDGET, type SafetyBudget } from './safety.js'; | ||
|
|
||
| const tmpDirs: string[] = []; | ||
| afterEach(() => { |
There was a problem hiding this comment.
REVIEWER: [low] This afterEach is registered at module top level, which under --experimental-test-isolation=none puts it on the shared root context — it runs after every test in the process, not after this file's 22. I verified it with a two-file probe: a root-level afterEach in file A fired after all three tests across A and B.
In practice it is harmless here (tmpDirs.splice(0) on an empty array ~2175 extra times), so this is hygiene, not a bug. But the CI comment in ci.yml documents that the flag's cost is exactly cross-file leakage and that each case was audited — this adds a new one, and a future edit to the hook body (anything that touches shared state or costs real time) would silently apply to the whole extractor suite.
Suggested improvement: move the afterEach registration inside the four describe blocks (or wrap the file in one outer describe), so the hook is scoped to this file's tests.
|
|
||
| expect(() => | ||
| enforceSourceBudget(root, { maxFileBytes: 4096, maxTotalBytes: 100, maxFiles: 1 }), | ||
| ).toThrow(); |
There was a problem hiding this comment.
REVIEWER: [low] This is the one throw assertion in the file with no message pattern — every other one pins the error (/symlink escapes clone root/, /file exceeds 999 bytes/, /tree exceeds 1199 bytes/, /more than 2 files/). As a negative control it is the assertion that most needs to be specific: its whole job is to prove the walk descended into vendor/, and a bare toThrow() is satisfied by any error at all, including an ENOENT from a future change to tree() or a throw raised before the walk ever reaches vendor/.
Note also that with maxFileBytes: 4096 and two 5000-byte files, the error that actually fires is file exceeds 4096 bytes — not the count/size blow-up the comment above the loop describes ("three files and ~10KB would be counted and both caps would blow"). The comment and the code disagree about which cap trips.
Suggested improvement: .toThrow(/file exceeds 4096 bytes/) (or drop the junk files to ~50 bytes each so the control trips maxFiles/maxTotalBytes, matching the comment, and assert that message).
| const root = tree({ 'src/app.ts': 'export const a = 1;' }); | ||
| // From `<root>/src`, `..` reaches `<root>`, and one more `..` reaches the shared tmp parent | ||
| // that also holds `outside`. | ||
| const relative = join('..', '..', outside.split('/').pop() as string, 'secret.txt'); |
There was a problem hiding this comment.
REVIEWER: [nitpick] outside.split('/').pop() hardcodes the POSIX separator while node:path is already imported in this file. basename(outside) says the same thing, needs no as string cast, and does not silently return the whole path on a platform where the separator differs. Same pattern at line 298 (${root}/), where sep is available.
Suggested improvement: import { join, dirname, basename, sep } from 'node:path' and use basename(outside) here.
|
|
||
| // lstat sees a link; realpath resolves it to somewhere that is not under the root. | ||
| expect(realpathSync(link)).toBe(outside); | ||
| expect(outside.startsWith(`${root}/`)).toBe(false); |
There was a problem hiding this comment.
REVIEWER: [low] The comment one line above says "lstat sees a link" but nothing here calls lstat. This test is the harness's stated defence against a runner that cannot create symlinks (the why-block's Limitations bullet names it by name: "A precondition test asserts the link is real"), and the assertion it actually makes — realpathSync(link) === outside — is satisfied by several non-symlink outcomes: a directory junction, a bind-style copy where the path happens to resolve, or any future fallback that reproduces the target path. It proves the path resolves; it does not prove the entry is a symbolic link.
Since enforceSourceBudget's whole branch is gated on st.isSymbolicLink(), that is the property the precondition needs to assert.
Suggested improvement: add expect(lstatSync(link).isSymbolicLink()).toBe(true) (import lstatSync), and consider asserting the same for the file-target link, since a directory symlink and a file symlink are created through different code paths on some platforms.
A verifier mutation-tested the extractor and one mutant survived in the worst place: `lstatSync(p)` → `statSync(p)` in `enforceSourceBudget`'s walk — making the symlink-escape guard FOLLOW symlinks, precisely what its own comment says it must never do — left the whole suite at 2175 passing, exit 0. The path-traversal guard had no test that failed when it broke. It matters more than a normal coverage gap. `safety.ts` runs inside the ephemeral container before extraction, against untrusted repositories, and is the only thing between a hostile repo's symlink and the container filesystem outside the clone — a container that also holds a GitHub token. It ships in the public MIT package whose selling point is that source never leaves the machine, so it is exactly the file an evaluating engineer reads to check whether we mean it. A silently-broken traversal guard undercuts the claim the package was published to make auditable. REAL TEMP TREES, NEVER A MOCKED `fs`. The guard's whole job is what the filesystem actually reports — `lstat` versus `stat`, what `realpath` resolves a link to, whether a dangling link throws — and the surviving mutant is a one-word difference between two of those calls. Every fixture is built on disk, including actual symlinks out of the root, one of them relative (`../../`), which is the shape a repository can really commit. `realpathSync` on the fixture root is load-bearing rather than tidy: on macOS `os.tmpdir()` sits under `/var`, itself a symlink to `/private/var`, and the guard compares resolved targets against a resolved root. A fixture keeping the unresolved path would make every in-tree symlink look like an escape — passing for the wrong reason, and still passing with the guard removed. TWELVE MUTANTS, TWELVE KILLS, against a baseline verified green first: lstat→stat (symbol-preserving, so the kill is behavioural and not an unresolved-symbol crash: 8 failures) · the escape rejection deleted · the separator dropped from the prefix match, which is the `/tmp/clone-12345-evil` bypass · maxFileBytes raised · each of the three budget comparisons inverted `>` → `>=`, which a single "too big throws" test cannot see · a SKIP_DIRS entry removed (twice — it must not throw, so the budget is the instrument) · every directory skipped · the dangling-link catch widened to swallow escapes. The skip-list block carries its own negative control: without one, a mutant that skipped EVERY directory would satisfy all six entries by counting nothing at all. The symlink block carries two: an in-tree link must still be ACCEPTED, and a dangling one ignored, or a guard that threw on every symlink would pass the escape tests and break every repo that vendors a link. And the harness asserts its own preconditions: that `symlinkSync` really produced a symlink (a runner without the privilege could silently make a copy, and every escape test would then pass over a plain file) and that the "outside" tree really is outside the root. Floor for @backthread/extractor: 2175 -> 2197.
Review mutation-tested this PR's own tests and found four of its guarantees unpinned. A change arguing "a guard with no failing test is not guarded" shipping four surviving mutants is the same defect one level up, so all four are closed. THE ORDER OF THE TWO CHECKS IS LOAD-BEARING, and nothing said so. `lstat` and the symlink branch run BEFORE `SKIP_DIRS`. Hoist the skip check above them and every existing test still passed — while an attacker gains six names they choose (`.git`, `node_modules`, `dist`, `build`, `.next`, `coverage`) that walk straight past the traversal guard. `ln -s /etc node_modules` is a one-line repository. Pinned now for every name on the list, not one of them. THE DEFAULT BUDGET WAS THE ONE PATH NOBODY TESTED, and it is the only one that ships: nothing calls `enforceSourceBudget` with a budget, so every cap comes from the default parameter — which could be widened to Infinity with the suite green. Covered now by one file a byte over the real `maxFileBytes`, called the way the container calls it, with its at-the-cap pair so it is the cap biting rather than "a big file throws". THE SELF-LINK BRANCH: deleting `target !== rootReal` makes `ln -s . self` fail the ingest of an otherwise innocent repository, silently. Repos really do commit that. AND THIS FILE'S OWN REASONING WAS WRONG. It claimed fixtures must pre-resolve their root or in-tree symlinks would look like escapes. False — `enforceSourceBudget` calls `realpathSync(root)` itself. Because every fixture pre-resolved anyway, replacing that call with `resolve(root)` survived the whole suite. A test resting on a false premise is a test nobody can maintain, so the note is corrected and an unresolved-root case now covers the mutant on both halves: an in-tree link still accepted, an escape still caught. Three smaller ones, all real: the only negative control was the file's only bare `toThrow()`, so it went green on any error including a broken fixture — it names its error now. A precondition described an `lstat` check in prose and never made the call, which is precisely the defect under test. And cleanup moved off a top-level `afterEach`: under `--experimental-test-isolation=none` that registers on the shared ROOT context and fires around every test in the run rather than this file's, so each test now owns and disposes of its own fixtures in a `finally` — no reasoning about the runner's context model required. Nine mutants re-run against a green baseline, nine kills, including all four survivors. Floor 2197 -> 2201.
Verification ran 36 mutants and found the resolved-root test platform-dependent. It relied on `os.tmpdir()` being reached through a symlink — true on macOS, where `/var` links to `/private/var`, and FALSE on the Linux runner CI actually uses, where it is plain `/tmp`. Measured: with the `realpathSync(root)` → `resolve(root)` mutant applied under a non-symlinked TMPDIR, the suite stayed at 26 of 26. The test was not flaky; it simply stopped testing anything. So the fixture now builds its OWN symlinked route to the root and passes that as the root. The symlink is ours rather than the operating system's, so the mutant dies everywhere. AND ONE GUARANTEE IS RECORDED AS UNPINNED RATHER THAN QUIETLY MISSED. The docstring promises a throw on the FIRST violation, and for a zip-bomb guard aborting mid-walk is most of the value — but hoisting the two count checks out of the loop leaves every test here green, because the same error is still raised, just later. No honest deterministic test exists: `readdirSync` order is unspecified, so every "the walk stopped before reaching X" fixture depends on which entry the filesystem returns first, and the only order-independent instrument is a timing assertion, which is flaky by construction. A guard that fails on a fast machine and passes on a slow one is worse than a stated gap. Pinning it needs an injectable readdir counter — a change to safety.ts's signature for testability, which is a decision for whoever needs the guarantee. Two further survivors are proved EQUIVALENT rather than missed (the `continue` after the symlink branch; `walk(resolve(root))` → `walk(root)`), and a third — `else if (st.isFile())` → `else` — is reachable only by a FIFO, socket or device node, none of which git can check out, and makes the guard stricter. All three are named in the file so the next mutation run does not re-discover them as findings.
4d3567e to
d46d00f
Compare

Why
A verifier mutation-tested the extractor while validating the CI rework, and one mutant survived in the worst possible place. In
packages/extractor/src/safety.ts, changelstatSync(p)tostatSync(p)inenforceSourceBudget's walk — i.e. make the symlink-escape guard follow symlinks, precisely the thing its own comment says it must never do — and the result is# tests 2175 / # pass 2175, exit 0. The path-traversal guard had no test that failed when it stopped working.This is not a normal coverage gap.
safety.tsruns inside the ephemeral container before extraction, against untrusted repositories, and is the only thing between a hostile repo's symlink and the container filesystem outside the clone — a container that also holds a GitHub token. It ships in the public MIT package whose selling point is that source never leaves the machine, so it is exactly the file an evaluating engineer reads to check whether we mean it.Test-only. No behaviour change, no version bump, nothing to publish.
Real temp trees, never a mocked
fsThe guard's whole job is what the filesystem actually reports —
lstatvsstat, whatrealpathresolves a link to, whether a dangling link throws — and the surviving mutant is a one-word difference between two of those calls. A mockedfswould test our beliefs about them instead of them. Every fixture is built on disk, including actual symlinks out of the root, one of them relative (../../), which is the shape a repository can really commit.realpathSyncon the fixture root is load-bearing, not tidiness. On macOSos.tmpdir()sits under/var, itself a symlink to/private/var, and the guard compares resolved link targets against a resolved root. A fixture keeping the unresolved path would make every in-tree symlink look like an escape — passing for the wrong reason, and still passing with the guard removed.Twelve mutants, twelve kills (baseline verified green first)
lstatSync→statSync— symbol-preserving (the import is added andlstatSynckept live), so the kill is behavioural rather than an unresolved-symbol crash/tmp/clone-12345-evilbypassmaxFileBytesraisedmaxFileBytescomparison inverted>→>=maxTotalBytescomparison invertedmaxFilescomparison invertedSKIP_DIRSlosescoverageSKIP_DIRSlosesnode_modulescatchwidened to swallow escapesThe three boundary pairs (at the cap → passes, one over → throws) are what catch the
>→>=inversions; a single "too big throws" test cannot see a one-character change.Negative controls, because each block could pass for the wrong reason
symlinkSyncreally produced a symlink (a runner without the privilege could silently make a copy, and every escape test would then pass over a plain file), and that the "outside" tree really is outside the root.Numbers
packages/extractor: 2175 → 2197 passing, floor raised to match.Reasoning
written by the agent in-session via
backthread/add-reasoning-to-prsDecisions
lstat→statmutant is run in a symbol-preserving form: the import is added andlstatSyncis kept referenced. Run naively it throws a ReferenceError and 18 tests fail, which proves nothing about the assertions. The symbol-preserving form kills 8 tests on behaviour.realpathSync. On macOS the temp directory sits under a symlinked/var, and the guard compares resolved targets to a resolved root. Without this every in-tree symlink looks like an escape, so the tests would pass with the guard deleted.Trade-offs
SKIP_DIRSentry does not throw. This makes them sensitive to the fixture sizes: if someone changes the fixture bytes without changing the caps, the tests can stop discriminating. Check: keep the cap just above the one real source file.Limitations
symlinkSynccould produce something else and every escape test would pass over a plain file. A precondition test asserts the link is real, so that case fails loudly instead of going green. Check: the test named "symlinkSync produces a real symlink".↳ generated by
backthread/add-reasoning-to-prs· an open-source Claude Code hook · edit or delete freely