{{ message }}
Repository navigation
Clean up the test suite: one test file per source, named fixtures, Go E2E runner - #192
Closed
nodeselector wants to merge 71 commits into
Closed
nodeselector wants to merge 71 commits into
nodeselector wants to merge 71 commits into
Conversation
Recover local fix fcdd43df96c77d5a376a3c27de8d03f07f6317c5 and retain both community normalization safeguards. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Parity now resolves recorded branch heads and fails closed when the recorded ref is inconclusive (#174). Stub fixtures ported from candidate commits 3eb0100 and 22e894c. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
CHECKOUT_SHA was the actions/checkout v6.0.2 commit while the fixtures record it under v4 and main. Recorded-ref tracing (#174) correctly rejects that; use the real v4.2.2 commit the stub tag list already names. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Restores the #167 same-run scenario with a recorded tag instead of swapping it for a branch, which only exercised the missing-head exit. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the original --verify bodies, adjusted for ref-qualified lookups and fail-closed inconclusive checks; replace the mode cross-products with one default/--no-fix test, and drop full-SHA and entry-ref cases that TestParityFinding_FullSHAKey and TestValidEntryRef already cover. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The backend resolver, not the runner, looks up and rejects lock entry refs; say so in comments, docs and the missing-ref finding detail. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Tags move, and the Actions backend accepts a pin whose commit is still in its tag's history. Every lock entry ref is now checked by reachability: a tag that lost the commit still blocks and an inconclusive check still fails closed, but a tag that moved ahead only warns under --verify. Dropping the separate tag-equality lookup also saves the extra REST commit call when a ref still points at its locked commit. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
An error-severity finding blocks the run, so it never renders as a warning. Under --verify the terminal now shows an exact tag that moved ahead of its locked commit, matching the JSON warning, and the inconclusive-check remediation names the Actions backend. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A ref like main~1 or v1@{1} is not a tag or branch name, but a commit-ish
lookup evaluates it as a revision. Validate names with git
check-ref-format rules, matching actions-lockfile's RefName: an entry ref
that fails makes the lockfile unreadable, and parity never builds a
lookup expression from one.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
--relock drops findings for recorded pins because it replaces them, but it also dropped them for pins it had just resolved, so it could write a lock that the next --verify rejects. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A full-SHA entry whose recorded tag or branch no longer vouches for it recommended --accept-moved, which then refused it. --accept-moved now accepts it as a warning and re-records a tag or branch that contains the commit, chosen by reverse lookup; a commit no branch contains still fails. Fresh full-SHA pins are no longer checked against a pre-planning ref. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The Actions backend resolver looks a recorded ref up as a tag before a branch, so reverse lookup now skips a containing branch whose name is also a tag. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A ref counts only if a tag or branch exists by exactly that name: GraphQL
also asks ref(qualifiedName:), and REST reads git/ref/{tags|heads}/<name>.
Like the Actions backend, names ending in -g plus four or more hex
digits are not accepted as recorded or uses: refs, so an entry with one
makes the lockfile unreadable and a uses: ref with one is never locked.
Adapted from 2134d14 and 93b2f4a.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
nodeselector
force-pushed
the
nodeselector-test-suite-cleanup
branch
from
October 20, 2026 03:51
d0d11a3 to
8b02826
Compare
nodeselector
changed the base branch from
nodeselector-174-trace-recorded-ref
to
nodeselector-transitive-lock-graphs
October 20, 2026 03:51
nodeselector
added this pull request to stack #191
October 20, 2026 03:51
--accept-moved now prints the old and new recorded ref when it keeps a full-SHA pin's commit but records a new tag or branch for it. A ref this run resolved that isn't a tag or branch is titled "Not a tag or branch"; "Ref no longer exists" is kept for a ref the lockfile recorded. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
nodeselector
force-pushed
the
nodeselector-transitive-lock-graphs
branch
from
October 20, 2026 03:58
b1a85f2 to
e3315a6
Compare
A tool failure now prints before the run log path instead of after it. Inconclusive checks and the Dependabot proxy abort show a short cause on the terminal, such as "could not check v4 history: GitHub returned 403 Forbidden"; --json and the run log keep the full error. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A full-SHA pin whose recorded tag or branch no longer exists was titled "Pinned commit not in upstream history". It is now "Ref no longer exists", like any recorded ref that is gone. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
nodeselector
force-pushed
the
nodeselector-test-suite-cleanup
branch
from
October 20, 2026 04:06
8b02826 to
2cca9d3
Compare
nodeselector
force-pushed
the
nodeselector-transitive-lock-graphs
branch
from
October 20, 2026 04:09
e3315a6 to
342b1a3
Compare
nodeselector
force-pushed
the
nodeselector-test-suite-cleanup
branch
from
October 20, 2026 04:14
2cca9d3 to
a0d9f34
Compare
Terminal warnings were grouped by dependency alone, so a full-SHA pin's sha-as-ref notice hid the --verify warning that its recorded exact tag moved ahead. Group by dependency and category. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved fixture-validation, E2E cache-isolation, reference-matching, and coverage-path defects require fixes.
1 open issue
What changed in this PR
Reorganizes tests into source-aligned files, centralizes named fixtures, and replaces the Ruby integration harness with Go E2E tests.
Changes:
- Consolidates tests and reusable HTTP fakes.
- Adds named pins, fixture validation, and test-layout guards.
- Updates scenario handling, Make targets, CI, and documentation.
Review effort: Max
Each internal test file is now named after the source file it exercises (foo.go -> foo_test.go). Pure moves: the same 367 test functions run before and after. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Command tests now live in the test file of the source that implements the behavior: runCheck wiring in run_test.go, root command flags, paths and the shared command harness in root_test.go, cooldown resolution in fresh_tag_test.go, the no-onboard gate in onboard_gate_test.go. Pure moves: the same 154 test functions run before and after. With no orphans left, TestEveryTestFileHasSource drops its allowlist. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
httpmock now owns the shapes command and API tests kept re-declaring: StubRepo for the batched resolve query, ParityIntact/ParityQuery, RoundTripFunc and CountingTransport. Command tests use workflowUsing for the standard one-job workflow and runCheckJSON/checkResult for the --json=valid,findings payload instead of hand-decoding it. The four reachability tests that differed only in the compare response are one table, TestCheck_RelockAncestry. Tenant-host tests get neutral names. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Go tests that spelled out actions/checkout and actions/setup-go release commits now reference testfixture pins, so each literal carries the release it came from and the fixture guard checks it. Adds the checkout v4.2.1 pin those tests use. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
lockfile, pin and pipeline tests each declared their own RepoIDs fake returning fixed IDs; testfixture.Metadata replaces all three. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
These files already asserted with testify everywhere else. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ServeHTTP and RegisterRepeated let a subprocess under test talk to the same stub registry the command tests use. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
test/e2e builds the binary once and runs every catalog scenario in a fresh git repo. Stub scenarios name a fake-GitHub profile (stub:) served by httpmock over TLS and run in go test ./...; live scenarios need E2E_LIVE=1. Custom assertions become declarative output_count and then_run, exit codes are matched exactly, and every no-token scenario is tagged stub so it guards merge. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
make test-integration, test-live, test-smoke, test-stub, and test-real now run go test ./test/e2e with E2E_LIVE / E2E_TAGS. CI no longer installs Ruby. The interactive shell and review/report tooling go with the harness. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
relock_bumps_stale_branch_ref restated TestCheck_Relock_BumpsMovedBranchRef; verify_local_conflict_relock restated TestValidateOutputFlags_VerifyConflicts (verify_conflict_verify_local still proves the real binary's usage exit code); no_workflows_summary folds into no_workflows. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The impostor-pin guard accepted any lookup error, so a 401, rate limit or outage passed it. It now requires a 404 or 422 and fails on anything inconclusive. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Three gaps in the runner: - Scenarios ran under the developer's HOME, so every run wrote its run log to the real user cache dir (where its retention GC could delete real logs), and live scenarios read the real user config file. Each scenario now gets its own HOME, XDG and gh config dirs; the unused GH_ACTIONS_LOCK_CACHE_DIR is gone. - The lockfile coherence check matched `uses:` by prefix, so a stale `actions/checkout@v4` entry passed against `@v4.2.2`. usesRef now requires the exact ref, with owner/repo folding case. - A relative E2E_COVERDIR resolved against each scenario's repository, silently dropping all coverage. It is now made absolute and created. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
TestCatalogSHAsAreNamedPins accepted any pin with the same SHA, so a real commit attached to another repository or recorded under the wrong ref passed. Bare `owner/repo@sha` uses must match the pin's repository, and expected-JSON nwo/ref/sha triples must match a pin exactly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Scenarios no longer inherit GH_HOST or enterprise tokens, so a developer's configured host can't redirect live runs. - The package builds on Windows: the binary gets its .exe suffix, USERPROFILE and AppData move into the scenario home, and the PTY runner is limited to platforms that have one. - lockfile_deps_cover_direct now also fails when a workflow uses a remote action its lockfile entry doesn't list, not only the reverse. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
nodeselector
force-pushed
the
nodeselector-test-suite-cleanup
branch
from
October 20, 2026 06:13
7b55c62 to
e4264b2
Compare
liveToken was a package-level sync.OnceValue, so whichever test called it first fixed the token for every later scenario; under -shuffle=on a test that set a fake GH_TOKEN made live scenarios authenticate with it. TestMain now resolves the token into Runner.Token and scenarioEnv reads it from there. The E2E make targets shuffle test order. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
nodeselector
force-pushed
the
nodeselector-transitive-lock-graphs
branch
from
October 20, 2026 07:00
8c589c0 to
931aebe
Compare
This was referenced Oct 20, 2026
nodeselector
force-pushed
the
nodeselector-transitive-lock-graphs
branch
from
October 21, 2026 00:13
f7ad446 to
729887b
Compare
nodeselector
removed this pull request from stack #191
October 21, 2026 00:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Planned layer 7/7 in the approved order: #197 → #198 → #190 → #195 → #196 → #193 → #192. Parent: #193; rebase and base retarget are pending. Native stack 191 retains its existing membership; the source-bound evidence below is historical, not qualification of the planned restack.
Stacked on #190 in the stack #186 → #190 → #192. This PR changes only tests, CI and docs; product behavior is unchanged.
What changed
internal/testfixturegives every real upstream commit the tests use a name and its source release.TestPinsAreInRecordedRefHistory(make test-fixtures, run by a new blockingfixture-guardjob and at the start ofintegration-live) proves each commit is in its recorded ref's history and that repository and owner IDs match. It caught theactions/setup-go@v5fixture, which pinned a v6.4.0 commit with checkout's repository ID.foo.go→foo_test.go); the 27 files that weren't are merged into the matching file, andTestEveryTestFileHasSourceenforces the rule.httpmocknow owns the fakes command and API tests kept re-declaring (StubRepo,ParityIntact/ParityQuery,RoundTripFunc,CountingTransport), andtestfixture.Metadatareplaces three copies of the same repository-ID fake.workflowUsingandrunCheckJSONinstead of repeating workflow bodies and JSON decoding, and tests that differed only in one stubbed response are now tables (TestCheck_RelockAncestry,TestShouldRetry).test/e2e(about 750 lines) replaces the 3,200-line Ruby harness, driving the compiled binary overtest/scenarios/catalog.ymlwith a TLS fake GitHub selected bystub:profiles. Stub scenarios now run ingo test ./....TestEveryScenarioRunsInCIrequires every scenario to be taggedstubor need a token.Reviewer guide
Reading order:
internal/testfixture/fixtures.goand its test: pins, the guard and the naming rule.internal/ghapi/httpmock/githubfixtures.go: the shared fakes.cmd/gh-actions-lock/root_test.go: the command harness. Inrun_test.go, the diff is mostly moved tests; the tables are the part to read.test/e2e/e2e.go, then thecatalog.ymldiff..github/workflows/test.yml,MakefileandAGENTS.md.Evidence against #190's head:
make testpasses.httpmockgoes from 35% to 68%)../actions, rejection of nested./actions, and checking against the recorded entry ref.E2E_LIVE=1 go test ./test/e2epasses. Before the Ruby was deleted, the Go runner matched it on all 103 scenarios.make test-fixturespasses.Expect
go test ./...takes about 20s longer, because the server-error scenario waits through the binary's retry backoff.make test-livenow runs the live scenarios; before, it selected none.github.com/creack/ptyis now a direct dependency, for the one interactive-prompt scenario.