Fail activation early for stale provenance workflows by Copilot · Pull Request #70348 · github/gh-aw · GitHub
Skip to content

Fail activation early for stale provenance workflows #70348

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pelikhan merged 6 commits into from
Oct 20, 2026
+138 −13
Merged

Fail activation early for stale provenance workflows #70348

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0b659e5
Initial plan
Copilot Oct 19, 2026
2c3ec8a
Fail activation on stale provenance workflows
Copilot Oct 19, 2026
22882ba
Merge remote-tracking branch 'origin/main' into copilot/fix-gh-aw-set…
Copilot Oct 19, 2026
bc85ee4
Scope provenance check to safe outputs
Copilot Oct 19, 2026
5ffbb94
Merge remote-tracking branch 'origin/main' into copilot/fix-gh-aw-set…
Copilot Oct 19, 2026
6d73d77
Handle missing provenance compatibility signal
Copilot Oct 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/patch-check-run-provenance-compatibility.md
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

32 changes: 32 additions & 0 deletions actions/setup/js/check_version_updates.cjs
View file
Open in desktop
109 changes: 96 additions & 13 deletions actions/setup/js/check_version_updates.test.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ describe("check_version_updates", () => {
vi.stubGlobal("fetch", mockFetch);

delete process.env.GH_AW_COMPILED_VERSION;
delete process.env.GH_AW_HAS_SAFE_OUTPUTS;
delete process.env.GH_AW_BLOCKED_VERSION_REPORT_AS_ISSUE;
delete process.env.GH_AW_WORKFLOW_NAME;
delete process.env.GITHUB_REPOSITORY;
Expand Down Expand Up @@ -169,6 +170,66 @@ describe("check_version_updates", () => {
expect(mockCore.info).toHaveBeenCalledWith(expect.stringContaining("Could not fetch update configuration"));
});

it("should reject workflows older than the provenance requirement when remote config is unavailable", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.88.7";
process.env.GH_AW_HAS_SAFE_OUTPUTS = "true";
mockFetchError(new Error("ECONNREFUSED"));
await runMain();
expect(mockFetch).toHaveBeenCalled();
expect(mockCore.setFailed).toHaveBeenCalledWith(expect.stringContaining("v0.91.10 or newer"));

@github-actions github-actions Bot Oct 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] Good coverage for the reject/accept boundary at v0.91.10, but there is no test for the exact boundary v0.91.9 (one patch below) failing and no test for the scenario above — a pre-v0.91.10 workflow without safe-outputs configured, which this check currently still blocks unnecessarily.

💡 Missing boundary/negative case 
it("should reject the patch version immediately below the provenance requirement", async () => {
  process.env.GH_AW_COMPILED_VERSION = "v0.91.9";
  mockFetchError(new Error("ECONNREFUSED"));
  await runMain();
  expect(mockCore.setFailed).toHaveBeenCalledWith(expect.stringContaining("v0.91.10 or newer"));
});

Exact-boundary tests read as a clearer specification of the cutoff than the current v0.88.7 example alone.

@copilot please address this.

Copilot AI Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the exact v0.91.9 rejection test and the older-version/no-safe-outputs regression case in bc85ee4e.

expect(mockCore.summary.addRaw).toHaveBeenCalledWith(expect.stringContaining("run `gh aw compile`"));
expect(mockCore.summary.write).toHaveBeenCalled();
});

it("should reject the version immediately below the provenance requirement", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.91.9";
process.env.GH_AW_HAS_SAFE_OUTPUTS = "true";
mockFetchError(new Error("ECONNREFUSED"));
await runMain();
expect(mockCore.setFailed).toHaveBeenCalledWith(expect.stringContaining("v0.91.10 or newer"));
});

it.each([
["missing", undefined],
["unknown", "maybe"],
])("should reject below-minimum workflows when the safe-output signal is %s", async (_signalState, signal) => {
process.env.GH_AW_COMPILED_VERSION = "v0.91.9";
if (signal === undefined) {
delete process.env.GH_AW_HAS_SAFE_OUTPUTS;
} else {
process.env.GH_AW_HAS_SAFE_OUTPUTS = signal;
}
mockFetchError(new Error("ECONNREFUSED"));
await runMain();
expect(mockCore.setFailed).toHaveBeenCalledWith(expect.stringContaining("v0.91.10 or newer"));
});

it("should not reject below-minimum workflows explicitly compiled without safe outputs when remote config is unavailable", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.88.7";
process.env.GH_AW_HAS_SAFE_OUTPUTS = "false";
mockFetchError(new Error("ECONNREFUSED"));
await runMain();
expect(mockCore.setFailed).not.toHaveBeenCalled();
expect(mockCore.info).toHaveBeenCalledWith(expect.stringContaining("Could not fetch update configuration"));
});

it("should accept the first compiler version that supplies safe-output provenance", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.91.10";
process.env.GH_AW_HAS_SAFE_OUTPUTS = "true";
mockFetchSuccess(JSON.stringify({}));
await runMain();
expect(mockCore.setFailed).not.toHaveBeenCalled();
expect(mockCore.info).toHaveBeenCalledWith(expect.stringContaining("Version check passed"));
});

it("should accept a provenance-compatible workflow when remote config is unavailable", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.91.10";
mockFetchError(new Error("ECONNREFUSED"));
await runMain();
expect(mockCore.setFailed).not.toHaveBeenCalled();
expect(mockCore.info).toHaveBeenCalledWith(expect.stringContaining("Could not fetch update configuration"));
});

it("should retry and succeed when fetch fails transiently", async () => {
process.env.GH_AW_COMPILED_VERSION = "v1.2.0";
// First call fails transiently; second call succeeds
Expand Down Expand Up @@ -389,6 +450,28 @@ describe("check_version_updates", () => {
expect(mockCore.setFailed).toHaveBeenCalledWith(expect.stringContaining("Blocked compile-agentic version"));
});

it("should report blocked old versions before enforcing the provenance minimum", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.85.3";
process.env.GH_AW_HAS_SAFE_OUTPUTS = "true";
mockFetchSuccess(JSON.stringify({ blockedVersions: ["v0.85.3"], minimumVersion: "" }));
const searchMock = vi.fn().mockResolvedValue({ data: { items: [] } });
const createMock = vi.fn().mockResolvedValue({ data: { number: 42, html_url: "https://github.com/owner/repo/issues/42" } });
global.github = {
rest: {
search: { issuesAndPullRequests: searchMock },
issues: { create: createMock, update: vi.fn() },
},
};

await runMain();

expect(mockCore.summary.addRaw).toHaveBeenCalledWith(expect.stringContaining("Blocked compile-agentic version"));
expect(mockCore.summary.addRaw).not.toHaveBeenCalledWith(expect.stringContaining("Incompatible compiled workflow"));
expect(searchMock).toHaveBeenCalled();
expect(createMock).toHaveBeenCalled();
expect(mockCore.setFailed).toHaveBeenCalledWith(expect.stringContaining("Blocked compile-agentic version"));
});

it("should update an existing blocked version issue when version is blocked", async () => {
process.env.GH_AW_COMPILED_VERSION = "v1.0.0";
mockFetchSuccess(JSON.stringify({ blockedVersions: ["v1.0.0"], minimumVersion: "" }));
Expand Down Expand Up @@ -493,11 +576,11 @@ describe("check_version_updates", () => {
// ---------------------------------------------------------------------------

it("should fail when version is below minimum", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.8.0";
process.env.GH_AW_COMPILED_VERSION = "v0.91.10";
mockFetchSuccess(JSON.stringify({ blockedVersions: [], minimumVersion: "v1.0.0" }));
await runMain();
expect(mockCore.setFailed).toHaveBeenCalledWith(expect.stringContaining("Outdated compile-agentic version"));
expect(mockCore.setFailed).toHaveBeenCalledWith(expect.stringContaining("v0.8.0"));
expect(mockCore.setFailed).toHaveBeenCalledWith(expect.stringContaining("v0.91.10"));
});

it("should fail when major version is older (v0.x.x vs min v1.0.0)", async () => {
Expand All @@ -522,14 +605,14 @@ describe("check_version_updates", () => {
});

it("should skip minimum check when minimumVersion is empty string", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.1.0";
process.env.GH_AW_COMPILED_VERSION = "v0.91.10";
mockFetchSuccess(JSON.stringify({ blockedVersions: [], minimumVersion: "" }));
await runMain();
expect(mockCore.setFailed).not.toHaveBeenCalled();
});

it("should skip minimum check when minimumVersion has no 'v' prefix (unknown format — ignore)", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.1.0";
process.env.GH_AW_COMPILED_VERSION = "v0.91.10";
// minimumVersion "2.0.0" has no v prefix — should be ignored
mockFetchSuccess(JSON.stringify({ blockedVersions: [], minimumVersion: "2.0.0" }));
await runMain();
Expand All @@ -538,15 +621,15 @@ describe("check_version_updates", () => {
});

it("should skip minimum check when minimumVersion is a garbage string", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.1.0";
process.env.GH_AW_COMPILED_VERSION = "v0.91.10";
mockFetchSuccess(JSON.stringify({ blockedVersions: [], minimumVersion: "latest" }));
await runMain();
expect(mockCore.setFailed).not.toHaveBeenCalled();
expect(mockCore.info).toHaveBeenCalledWith(expect.stringContaining("Version check passed"));
});

it("should write summary when version is below minimum", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.8.0";
process.env.GH_AW_COMPILED_VERSION = "v0.91.10";
mockFetchSuccess(JSON.stringify({ blockedVersions: [], minimumVersion: "v1.0.0" }));
await runMain();
expect(mockCore.summary.addRaw).toHaveBeenCalledWith(expect.stringContaining("Outdated compile-agentic version"));
Expand Down Expand Up @@ -605,12 +688,12 @@ describe("check_version_updates", () => {
// ---------------------------------------------------------------------------

it("should warn when version is below minRecommendedVersion", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.9.0";
process.env.GH_AW_COMPILED_VERSION = "v0.91.10";
mockFetchSuccess(JSON.stringify({ blockedVersions: [], minimumVersion: "", minRecommendedVersion: "v1.0.0" }));
await runMain();
expect(mockCore.setFailed).not.toHaveBeenCalled();
expect(mockCore.warning).toHaveBeenCalledWith(expect.stringContaining("Recommended upgrade"));
expect(mockCore.warning).toHaveBeenCalledWith(expect.stringContaining("v0.9.0"));
expect(mockCore.warning).toHaveBeenCalledWith(expect.stringContaining("v0.91.10"));
expect(mockCore.warning).toHaveBeenCalledWith(expect.stringContaining("v1.0.0"));
});

Expand All @@ -631,39 +714,39 @@ describe("check_version_updates", () => {
});

it("should skip recommended check when minRecommendedVersion is empty string", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.5.0";
process.env.GH_AW_COMPILED_VERSION = "v0.91.10";
mockFetchSuccess(JSON.stringify({ blockedVersions: [], minimumVersion: "", minRecommendedVersion: "" }));
await runMain();
expect(mockCore.setFailed).not.toHaveBeenCalled();
expect(mockCore.warning).not.toHaveBeenCalled();
});

it("should skip recommended check when minRecommendedVersion has no 'v' prefix (unknown format — ignore)", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.5.0";
process.env.GH_AW_COMPILED_VERSION = "v0.91.10";
mockFetchSuccess(JSON.stringify({ blockedVersions: [], minimumVersion: "", minRecommendedVersion: "2.0.0" }));
await runMain();
expect(mockCore.setFailed).not.toHaveBeenCalled();
expect(mockCore.warning).not.toHaveBeenCalled();
});

it("should skip recommended check when minRecommendedVersion is a garbage string", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.5.0";
process.env.GH_AW_COMPILED_VERSION = "v0.91.10";
mockFetchSuccess(JSON.stringify({ blockedVersions: [], minimumVersion: "", minRecommendedVersion: "latest" }));
await runMain();
expect(mockCore.setFailed).not.toHaveBeenCalled();
expect(mockCore.warning).not.toHaveBeenCalled();
});

it("should skip recommended check when minRecommendedVersion is null (treated as missing)", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.5.0";
process.env.GH_AW_COMPILED_VERSION = "v0.91.10";
mockFetchSuccess(JSON.stringify({ blockedVersions: [], minimumVersion: "", minRecommendedVersion: null }));
await runMain();
expect(mockCore.setFailed).not.toHaveBeenCalled();
expect(mockCore.warning).not.toHaveBeenCalled();
});

it("should fail hard (not just warn) when version is also below minimumVersion", async () => {
process.env.GH_AW_COMPILED_VERSION = "v0.8.0";
process.env.GH_AW_COMPILED_VERSION = "v0.91.10";
mockFetchSuccess(JSON.stringify({ blockedVersions: [], minimumVersion: "v1.0.0", minRecommendedVersion: "v1.0.0" }));
await runMain();
// Should fail, not just warn
Expand Down
2 changes: 2 additions & 0 deletions docs/src/content/docs/reference/releases.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,8 @@ At runtime, the activation job fetches `.github/aw/compat.json` and compares the

This means that if you compiled a workflow months ago with an older version of `gh aw`, you may see a warning (or failure) asking you to recompile with a newer version. Run `gh aw upgrade` to bring all workflows up to date.

Setup action dependency upgrades can also introduce runtime requirements that older generated workflows do not meet. If activation reports that a workflow is too old for safe-output publishing, update `gh-aw`, run `gh aw compile`, and commit the regenerated workflow files. Updating only the pinned setup action is not sufficient; older workflows must pass the publishing job's check-run context for execution provenance. This requirement is enforced by the newer setup runtime and does not retroactively block workflows that remain pinned to an older runtime.

## How Lock Files Are Pinned

When `gh aw compile` generates a `.lock.yml`, it pins every GitHub Actions reference to a commit SHA:
Expand Down
1 change: 1 addition & 0 deletions pkg/workflow/compiler_activation_steps.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,7 @@ func (c *Compiler) addActivationVersionCheckStep(ctx *activationJobBuildContext)
ctx.steps = append(ctx.steps, fmt.Sprintf(" uses: %s\n", getCachedActionPin("actions/github-script", ctx.data)))
ctx.steps = append(ctx.steps, " env:\n")
ctx.steps = append(ctx.steps, fmt.Sprintf(" GH_AW_COMPILED_VERSION: \"%s\"\n", c.version))
ctx.steps = append(ctx.steps, fmt.Sprintf(" GH_AW_HAS_SAFE_OUTPUTS: \"%t\"\n", ctx.data.SafeOutputs != nil))
ctx.steps = append(ctx.steps, buildTemplatableBoolEnvVar("GH_AW_BLOCKED_VERSION_REPORT_AS_ISSUE", activationBlockedVersionReportAsIssueValue(ctx))...)
ctx.steps = append(ctx.steps, fmt.Sprintf(" GH_AW_WORKFLOW_NAME: %q\n", ctx.data.Name))
ctx.steps = append(ctx.steps, " with:\n")
Expand Down
2 changes: 2 additions & 0 deletions pkg/workflow/compiler_activation_steps_test.go
View file
Open in desktop
Loading