The symlink-escape guard now fails when it stops working by bogatyrjov1 · Pull Request #172 · backthread/backthread · GitHub
Skip to content

The symlink-escape guard now fails when it stops working - #172

Merged
bogatyrjov1 merged 3 commits into
mainfrom
arp-1678-extractor-symlink-guard-tests
Aug 20, 2026
Merged

bogatyrjov1 merged 3 commits into
mainfrom
arp-1678-extractor-symlink-guard-tests

Conversation

@bogatyrjov1

@bogatyrjov1 bogatyrjov1 commented Aug 19, 2026 •

Copy link
Copy Markdown
Contributor

Why

A verifier mutation-tested the extractor while validating the CI rework, and one mutant survived in the worst possible place. In packages/extractor/src/safety.ts, change lstatSync(p) to statSync(p) in enforceSourceBudget's walk — i.e. make the symlink-escape guard follow symlinks, precisely the thing its own comment says it must never do — and the result is # tests 2175 / # pass 2175, exit 0. The path-traversal guard had no test that failed when it stopped working.

This is not a normal coverage gap. safety.ts runs inside the ephemeral container before extraction, against untrusted repositories, and is the only thing between a hostile repo's symlink and the container filesystem outside the clone — a container that also holds a GitHub token. It ships in the public MIT package whose selling point is that source never leaves the machine, so it is exactly the file an evaluating engineer reads to check whether we mean it.

Test-only. No behaviour change, no version bump, nothing to publish.

Real temp trees, never a mocked fs

The guard's whole job is what the filesystem actually reports — lstat vs stat, what realpath resolves a link to, whether a dangling link throws — and the surviving mutant is a one-word difference between two of those calls. A mocked fs would test our beliefs about them instead of them. Every fixture is built on disk, including actual symlinks out of the root, one of them relative (../../), which is the shape a repository can really commit.

realpathSync on the fixture root is load-bearing, not tidiness. On macOS os.tmpdir() sits under /var, itself a symlink to /private/var, and the guard compares resolved link targets against a resolved root. A fixture keeping the unresolved path would make every in-tree symlink look like an escape — passing for the wrong reason, and still passing with the guard removed.

Twelve mutants, twelve kills (baseline verified green first)

Mutant Failures
lstatSync → statSync — symbol-preserving (the import is added and lstatSync kept live), so the kill is behavioural rather than an unresolved-symbol crash 8
the "resolves outside the clone root" rejection deleted 5
the separator dropped from the prefix match — the /tmp/clone-12345-evil bypass 1
maxFileBytes raised 1
maxFileBytes comparison inverted > → >= 1
maxTotalBytes comparison inverted 1
maxFiles comparison inverted 10
SKIP_DIRS loses coverage 1
SKIP_DIRS loses node_modules 2
every directory skipped 5
the dangling-link catch widened to swallow escapes 5

The three boundary pairs (at the cap → passes, one over → throws) are what catch the > → >= inversions; a single "too big throws" test cannot see a one-character change.

Negative controls, because each block could pass for the wrong reason

  • The skip list is not a "does it throw" test — removing an entry makes the walk descend, and the only observable consequence is that more is counted. The budget is the instrument. Without a control, a mutant that skipped every directory would satisfy all six entries by counting nothing at all, so there is a test that a directory not on the list is walked.
  • The symlink block would be satisfied by a guard that threw on every symlink — which would break every repo that vendors one. So an in-tree link must still be accepted, and a dangling one ignored.
  • The harness asserts its own preconditions: that symlinkSync really produced a symlink (a runner without the privilege could silently make a copy, and every escape test would then pass over a plain file), and that the "outside" tree really is outside the root.

Numbers

packages/extractor: 2175 → 2197 passing, floor raised to match.

Reasoning

written by the agent in-session via backthread/add-reasoning-to-prs

Decisions

  • The lstat → stat mutant is run in a symbol-preserving form: the import is added and lstatSync is kept referenced. Run naively it throws a ReferenceError and 18 tests fail, which proves nothing about the assertions. The symbol-preserving form kills 8 tests on behaviour.
  • Fixture roots go through realpathSync. On macOS the temp directory sits under a symlinked /var, and the guard compares resolved targets to a resolved root. Without this every in-tree symlink looks like an escape, so the tests would pass with the guard deleted.

Trade-offs

  • The skip-list tests assert through the budget rather than by inspecting a file list, because removing a SKIP_DIRS entry does not throw. This makes them sensitive to the fixture sizes: if someone changes the fixture bytes without changing the caps, the tests can stop discriminating. Check: keep the cap just above the one real source file.

Limitations

  • Assumes the test runner can create symlinks. On a filesystem or CI runner without that privilege symlinkSync could produce something else and every escape test would pass over a plain file. A precondition test asserts the link is real, so that case fails loudly instead of going green. Check: the test named "symlinkSync produces a real symlink".
    ↳ generated by backthread/add-reasoning-to-prs · an open-source Claude Code hook · edit or delete freely

Comment thread packages/extractor/src/safety.test.ts Outdated
* would pass for the wrong reason and would keep passing with the guard removed.
*/
function tree(files: Record<string, string>): string {
const dir = realpathSync(mkdtempSync(join(tmpdir(), 'bt-safety-')));

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REVIEWER: [medium] I ran this mutant and it survives: change const rootReal = realpathSync(root) to const rootReal = resolve(root) in safety.ts and the file still reports 22 pass / 0 fail. That is the one line that makes an unresolved caller path work, and nothing here pins it.

The reason it is unreachable is this realpathSync on the fixture root, and the justification given for it (repeated in the why-block Decisions) is not what the code does. I probed it on this machine:

tmpdir()         = /var/folders/3y/.../T
raw fixture root = /var/folders/3y/.../T/bt-probe-Ehbt5V
realpath root    = /private/var/folders/3y/.../T/bt-probe-Ehbt5V
root is a symlinked path? true
UNRESOLVED root + in-tree symlink => ACCEPTED (no throw)

enforceSourceBudget resolves the root itself (realpathSync(root)) before comparing, so an unresolved /var/... root does not make in-tree symlinks look like escapes. The claim "without this every in-tree symlink looks like an escape, so the tests would pass with the guard deleted" is false — the accept test would have failed, not passed.

What realpathSync here is actually load-bearing for is the harness's own string assertions (line 297 expect(realpathSync(link)).toBe(outside), line 298 outside.startsWith(root + '/'), and the ${root}-evil sibling). That is a fine reason to keep it — but it normalises away the exact production shape the guard's own realpathSync exists for, and the comment sells the opposite.

Suggested improvement: keep the resolved root for the existing fixtures, but add one test that passes the unresolved path (mkdtempSync(...) without realpathSync) containing an in-tree symlink and asserts .not.toThrow(). That test kills the resolve(root) mutant. Then rewrite this docblock (and the Decisions bullet) to say what is true: the resolved root is needed by the harness's path comparisons, not by the guard.

// stepped over, and the only observable consequence is that MORE is counted. So the budget is
// the instrument: a cap that fits the source files alone must survive the presence of a large
// skipped directory, and must not survive that directory being walked.
for (const skipped of ['.git', 'node_modules', 'dist', 'build', '.next', 'coverage']) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REVIEWER: [medium] These six tests, plus the negative control below, do not constrain where the skip check sits relative to the symlink branch — and that ordering is the traversal guard. I ran the mutant: hoist if (SKIP_DIRS.has(name)) continue; to the top of the for loop, above lstatSync(p) (a plausible "don't lstat what we skip anyway" optimisation), and the file still reports 22 pass / 0 fail.

Under that mutant a hostile repo commits ln -s /etc node_modules (or .git, dist, build, .next, coverage — six attacker-choosable names, all conventionally gitignored so nobody looks) and the escape check never runs on it. For a suite whose stated premise is "removing the guarantee turns something red", this is the same class of hole the PR was opened to close.

Suggested improvement: add one test — a symlink named a skipped directory pointing outside the root — asserting toThrow(/symlink escapes clone root/). One case (node_modules → outsideTree()) is enough to pin the ordering; a loop over all six is cheap if you want it.

Comment thread packages/extractor/src/safety.test.ts Outdated
// A budget nobody pins can be "tuned" to infinity in a one-line diff. These three numbers
// are the whole guard when no caller passes an override, and `enforceSourceBudget`'s
// default parameter is what the container actually uses.
expect(DEFAULT_BUDGET).toEqual({

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REVIEWER: [medium] This test's own comment says the point is that "a budget nobody pins can be 'tuned' to infinity in a one-line diff" and that "enforceSourceBudget's default parameter is what the container actually uses" — but it only asserts the shape of the exported constant. No test ever calls enforceSourceBudget(root) with the budget omitted, so nothing connects the constant to the default parameter.

I ran the mutant: change the signature to budget: SafetyBudget = { maxFileBytes: Infinity, maxTotalBytes: Infinity, maxFiles: Infinity }, leaving DEFAULT_BUDGET exported and untouched — 22 pass / 0 fail. That is precisely the one-line tune-to-infinity the test claims to prevent, and it is the path that matters most: enforceSourceBudget has no in-repo caller (only index.ts re-exports it), so the container almost certainly calls it with no second argument.

Suggested improvement: add a test that exercises the default parameter — e.g. build a tree with one file over DEFAULT_BUDGET.maxFileBytes and assert enforceSourceBudget(root) (no budget) throws /file exceeds 8388608 bytes/. A cheaper alternative that also kills the mutant is expect(enforceSourceBudget.length).toBe(1) plus a no-arg call on a fixture built from DEFAULT_BUDGET's own numbers, but the real-tree version is the honest one.

Comment thread packages/extractor/src/safety.test.ts Outdated
// PROMISE 1 + 2 — SYMLINKS ARE NOT FOLLOWED, AND ONE THAT ESCAPES IS REJECTED.

describe('the path-traversal guard', () => {
test('rejects a symlink whose target resolves outside the clone root', () => {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REVIEWER: [low] The guard's condition has two clauses — target !== rootReal && !target.startsWith(rootReal + sep) — and this block only pins the second. I ran the mutant: delete target !== rootReal && and the file still reports 22 pass / 0 fail.

The surviving behaviour is a false rejection, not a bypass, so it is not a security hole — but a repo that commits ln -s . self (or any link resolving to the clone root itself) would then fail its whole ingest with "symlink escapes clone root", and nothing here would catch the regression. The -evil sibling test at line 152 covers the separator; this clause has no counterpart.

Suggested improvement: add a test that a symlink pointing at the clone root itself is accepted — symlinkSync(root, join(root, 'self')), expect(...).not.toThrow().

Comment thread packages/extractor/src/safety.test.ts Outdated
import { enforceSourceBudget, DEFAULT_BUDGET, type SafetyBudget } from './safety.js';

const tmpDirs: string[] = [];
afterEach(() => {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REVIEWER: [low] This afterEach is registered at module top level, which under --experimental-test-isolation=none puts it on the shared root context — it runs after every test in the process, not after this file's 22. I verified it with a two-file probe: a root-level afterEach in file A fired after all three tests across A and B.

In practice it is harmless here (tmpDirs.splice(0) on an empty array ~2175 extra times), so this is hygiene, not a bug. But the CI comment in ci.yml documents that the flag's cost is exactly cross-file leakage and that each case was audited — this adds a new one, and a future edit to the hook body (anything that touches shared state or costs real time) would silently apply to the whole extractor suite.

Suggested improvement: move the afterEach registration inside the four describe blocks (or wrap the file in one outer describe), so the hook is scoped to this file's tests.

Comment thread packages/extractor/src/safety.test.ts Outdated

expect(() =>
enforceSourceBudget(root, { maxFileBytes: 4096, maxTotalBytes: 100, maxFiles: 1 }),
).toThrow();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REVIEWER: [low] This is the one throw assertion in the file with no message pattern — every other one pins the error (/symlink escapes clone root/, /file exceeds 999 bytes/, /tree exceeds 1199 bytes/, /more than 2 files/). As a negative control it is the assertion that most needs to be specific: its whole job is to prove the walk descended into vendor/, and a bare toThrow() is satisfied by any error at all, including an ENOENT from a future change to tree() or a throw raised before the walk ever reaches vendor/.

Note also that with maxFileBytes: 4096 and two 5000-byte files, the error that actually fires is file exceeds 4096 bytes — not the count/size blow-up the comment above the loop describes ("three files and ~10KB would be counted and both caps would blow"). The comment and the code disagree about which cap trips.

Suggested improvement: .toThrow(/file exceeds 4096 bytes/) (or drop the junk files to ~50 bytes each so the control trips maxFiles/maxTotalBytes, matching the comment, and assert that message).

Comment thread packages/extractor/src/safety.test.ts Outdated
const root = tree({ 'src/app.ts': 'export const a = 1;' });
// From `<root>/src`, `..` reaches `<root>`, and one more `..` reaches the shared tmp parent
// that also holds `outside`.
const relative = join('..', '..', outside.split('/').pop() as string, 'secret.txt');

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REVIEWER: [nitpick] outside.split('/').pop() hardcodes the POSIX separator while node:path is already imported in this file. basename(outside) says the same thing, needs no as string cast, and does not silently return the whole path on a platform where the separator differs. Same pattern at line 298 (${root}/), where sep is available.

Suggested improvement: import { join, dirname, basename, sep } from 'node:path' and use basename(outside) here.

Comment thread packages/extractor/src/safety.test.ts Outdated

// lstat sees a link; realpath resolves it to somewhere that is not under the root.
expect(realpathSync(link)).toBe(outside);
expect(outside.startsWith(`${root}/`)).toBe(false);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REVIEWER: [low] The comment one line above says "lstat sees a link" but nothing here calls lstat. This test is the harness's stated defence against a runner that cannot create symlinks (the why-block's Limitations bullet names it by name: "A precondition test asserts the link is real"), and the assertion it actually makes — realpathSync(link) === outside — is satisfied by several non-symlink outcomes: a directory junction, a bind-style copy where the path happens to resolve, or any future fallback that reproduces the target path. It proves the path resolves; it does not prove the entry is a symbolic link.

Since enforceSourceBudget's whole branch is gated on st.isSymbolicLink(), that is the property the precondition needs to assert.

Suggested improvement: add expect(lstatSync(link).isSymbolicLink()).toBe(true) (import lstatSync), and consider asserting the same for the file-target link, since a directory symlink and a file symlink are created through different code paths on some platforms.

@bogatyrjov1

Copy link
Copy Markdown
Contributor Author

A verifier mutation-tested the extractor and one mutant survived in the worst
place: `lstatSync(p)` → `statSync(p)` in `enforceSourceBudget`'s walk — making
the symlink-escape guard FOLLOW symlinks, precisely what its own comment says it
must never do — left the whole suite at 2175 passing, exit 0. The path-traversal
guard had no test that failed when it broke.

It matters more than a normal coverage gap. `safety.ts` runs inside the ephemeral
container before extraction, against untrusted repositories, and is the only
thing between a hostile repo's symlink and the container filesystem outside the
clone — a container that also holds a GitHub token. It ships in the public MIT
package whose selling point is that source never leaves the machine, so it is
exactly the file an evaluating engineer reads to check whether we mean it. A
silently-broken traversal guard undercuts the claim the package was published to
make auditable.

REAL TEMP TREES, NEVER A MOCKED `fs`. The guard's whole job is what the
filesystem actually reports — `lstat` versus `stat`, what `realpath` resolves a
link to, whether a dangling link throws — and the surviving mutant is a one-word
difference between two of those calls. Every fixture is built on disk, including
actual symlinks out of the root, one of them relative (`../../`), which is the
shape a repository can really commit.

`realpathSync` on the fixture root is load-bearing rather than tidy: on macOS
`os.tmpdir()` sits under `/var`, itself a symlink to `/private/var`, and the
guard compares resolved targets against a resolved root. A fixture keeping the
unresolved path would make every in-tree symlink look like an escape — passing
for the wrong reason, and still passing with the guard removed.

TWELVE MUTANTS, TWELVE KILLS, against a baseline verified green first:

  lstat→stat (symbol-preserving, so the kill is behavioural and not an
  unresolved-symbol crash: 8 failures) · the escape rejection deleted · the
  separator dropped from the prefix match, which is the `/tmp/clone-12345-evil`
  bypass · maxFileBytes raised · each of the three budget comparisons inverted
  `>` → `>=`, which a single "too big throws" test cannot see · a SKIP_DIRS entry
  removed (twice — it must not throw, so the budget is the instrument) · every
  directory skipped · the dangling-link catch widened to swallow escapes.

The skip-list block carries its own negative control: without one, a mutant that
skipped EVERY directory would satisfy all six entries by counting nothing at all.
The symlink block carries two: an in-tree link must still be ACCEPTED, and a
dangling one ignored, or a guard that threw on every symlink would pass the
escape tests and break every repo that vendors a link.

And the harness asserts its own preconditions: that `symlinkSync` really produced
a symlink (a runner without the privilege could silently make a copy, and every
escape test would then pass over a plain file) and that the "outside" tree really
is outside the root.

Floor for @backthread/extractor: 2175 -> 2197.
Review mutation-tested this PR's own tests and found four of its guarantees
unpinned. A change arguing "a guard with no failing test is not guarded" shipping
four surviving mutants is the same defect one level up, so all four are closed.

THE ORDER OF THE TWO CHECKS IS LOAD-BEARING, and nothing said so. `lstat` and the
symlink branch run BEFORE `SKIP_DIRS`. Hoist the skip check above them and every
existing test still passed — while an attacker gains six names they choose
(`.git`, `node_modules`, `dist`, `build`, `.next`, `coverage`) that walk straight
past the traversal guard. `ln -s /etc node_modules` is a one-line repository.
Pinned now for every name on the list, not one of them.

THE DEFAULT BUDGET WAS THE ONE PATH NOBODY TESTED, and it is the only one that
ships: nothing calls `enforceSourceBudget` with a budget, so every cap comes from
the default parameter — which could be widened to Infinity with the suite green.
Covered now by one file a byte over the real `maxFileBytes`, called the way the
container calls it, with its at-the-cap pair so it is the cap biting rather than
"a big file throws".

THE SELF-LINK BRANCH: deleting `target !== rootReal` makes `ln -s . self` fail the
ingest of an otherwise innocent repository, silently. Repos really do commit that.

AND THIS FILE'S OWN REASONING WAS WRONG. It claimed fixtures must pre-resolve
their root or in-tree symlinks would look like escapes. False —
`enforceSourceBudget` calls `realpathSync(root)` itself. Because every fixture
pre-resolved anyway, replacing that call with `resolve(root)` survived the whole
suite. A test resting on a false premise is a test nobody can maintain, so the
note is corrected and an unresolved-root case now covers the mutant on both
halves: an in-tree link still accepted, an escape still caught.

Three smaller ones, all real: the only negative control was the file's only bare
`toThrow()`, so it went green on any error including a broken fixture — it names
its error now. A precondition described an `lstat` check in prose and never made
the call, which is precisely the defect under test. And cleanup moved off a
top-level `afterEach`: under `--experimental-test-isolation=none` that registers
on the shared ROOT context and fires around every test in the run rather than
this file's, so each test now owns and disposes of its own fixtures in a
`finally` — no reasoning about the runner's context model required.

Nine mutants re-run against a green baseline, nine kills, including all four
survivors. Floor 2197 -> 2201.
Verification ran 36 mutants and found the resolved-root test platform-dependent.
It relied on `os.tmpdir()` being reached through a symlink — true on macOS,
where `/var` links to `/private/var`, and FALSE on the Linux runner CI actually
uses, where it is plain `/tmp`. Measured: with the `realpathSync(root)` →
`resolve(root)` mutant applied under a non-symlinked TMPDIR, the suite stayed at
26 of 26. The test was not flaky; it simply stopped testing anything.

So the fixture now builds its OWN symlinked route to the root and passes that as
the root. The symlink is ours rather than the operating system's, so the mutant
dies everywhere.

AND ONE GUARANTEE IS RECORDED AS UNPINNED RATHER THAN QUIETLY MISSED. The
docstring promises a throw on the FIRST violation, and for a zip-bomb guard
aborting mid-walk is most of the value — but hoisting the two count checks out of
the loop leaves every test here green, because the same error is still raised,
just later. No honest deterministic test exists: `readdirSync` order is
unspecified, so every "the walk stopped before reaching X" fixture depends on
which entry the filesystem returns first, and the only order-independent
instrument is a timing assertion, which is flaky by construction. A guard that
fails on a fast machine and passes on a slow one is worse than a stated gap.
Pinning it needs an injectable readdir counter — a change to safety.ts's
signature for testability, which is a decision for whoever needs the guarantee.

Two further survivors are proved EQUIVALENT rather than missed (the `continue`
after the symlink branch; `walk(resolve(root))` → `walk(root)`), and a third —
`else if (st.isFile())` → `else` — is reachable only by a FIFO, socket or device
node, none of which git can check out, and makes the guard stricter. All three
are named in the file so the next mutation run does not re-discover them as
findings.
@bogatyrjov1
bogatyrjov1 force-pushed the arp-1678-extractor-symlink-guard-tests branch from 4d3567e to d46d00f Compare August 20, 2026 08:08
@bogatyrjov1
bogatyrjov1 merged commit 8607837 into main Aug 20, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants