Gate dry-run scans on added actionable findings (#69193) · github/gh-aw@7573428 · GitHub
Skip to content

Commit 7573428

Browse files
pelikhanCopilotCopilot
authored
Gate dry-run scans on added actionable findings (#69193)
* Gate dry-run scans on added actionable findings Compare scanner findings against pre-compilation lock files, ignore low-severity findings, and apply the shared dry-run zizmor secrets-outside-env filter. Preserve scanner execution failures and normal compilation policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix dry-run scanner review findings Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Apply remaining changes Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Refresh validation issue schemas after main merge Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Verify baseline and current scans in dry-run report test Assert both ShellCheck passes, their input paths, strict mode, and lock emission ordering after resource baselining added a pre-compilation scan. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
1 parent 836ec2b commit 7573428

26 files changed

Lines changed: 1316 additions & 108 deletions

‎.github/aw/debug-agentic-workflow.md‎

Lines changed: 23 additions & 10 deletions

‎cmd/gh-aw/main.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -810,8 +810,8 @@ func configureCompileBuildFlags() {
810810
compileCmd.Flags().Bool("no-emit", false, "Validate workflow without generating lock files")
811811
compileCmd.Flags().Bool("purge", false, "Delete .lock.yml files that were not regenerated during compilation (only when no specific files are provided)")
812812
compileCmd.Flags().Bool("strict", false, "Override frontmatter to enforce strict mode validation for all workflows (enforces action pinning, network config, safe-outputs, disallows write permissions and deprecated fields). Note: Workflows default to strict mode unless frontmatter sets strict: false")
813-
compileCmd.Flags().Bool("dry-run", false, "Compile with strict validation, shellcheck, and model checks; Docker-based scanners are opt-in; report gate/scanner coverage; disable compiler-managed GitHub mutations, stage safe outputs, and record dry_run in aw_info.json; custom scripts and external MCP effects remain unverified; does not authorize execution, upload, or dispatch workflows")
814-
compileCmd.Flags().Bool("allow-experimental", false, "Accept experimental-feature notices during dry-run compilation; notices remain visible and all other warnings and scanner failures remain fatal")
813+
compileCmd.Flags().Bool("dry-run", false, "Compile with strict validation, shellcheck, and model checks; gate added medium-or-higher scanner findings against pre-compilation locks; ignore low severity and zizmor secrets-outside-env; Docker-based scanners are opt-in; report gate/scanner coverage; disable compiler-managed GitHub mutations, stage safe outputs, and record dry_run in aw_info.json; custom scripts and external MCP effects remain unverified; does not authorize execution, upload, or dispatch workflows")
814+
compileCmd.Flags().Bool("allow-experimental", false, "Record explicit acceptance of nonblocking experimental-feature notices during dry-run compilation; validation errors, added actionable findings, and scanner failures remain fatal")
815815
compileCmd.Flags().String("environment", "", "Override the environment on every generated job; fails for reusable-workflow caller jobs, which cannot declare an environment")
816816
compileCmd.Flags().Bool("require-self-hosted-runners", false, "Fail compilation unless every generated workflow job selects a self-hosted runner")
817817
compileCmd.Flags().Bool("trial", false, "Enable trial mode compilation (modifies workflows for trial execution)")

‎docs/src/content/docs/reference/compilation-process.md‎

Lines changed: 27 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -229,7 +229,7 @@ Pre-activation runs gating checks sequentially before any AI execution. Any fail
229229
| `gh aw compile my-workflow` | Compile specific workflow |
230230
| `gh aw compile --verbose` | Enable verbose output |
231231
| `gh aw compile --strict` | Enhanced security validation |
232-
| `gh aw compile my-workflow --dry-run` | Compile for development testing with staging, mandatory validation/shellcheck/model checks, and warnings as errors; Docker scanners are opt-in |
232+
| `gh aw compile my-workflow --dry-run` | Compile with staging and mandatory validation/shellcheck/model checks; gate added medium-or-higher scanner findings against pre-compilation locks; Docker scanners are opt-in |
233233
| `gh aw compile my-workflow --environment gh-aw-debug` | Replace the environment on every generated job |
234234
| `gh aw compile --no-emit` | Validate without generating files |
235235
| `gh aw compile --actionlint --zizmor --poutine --grant` | Run security scanners |
@@ -260,9 +260,22 @@ pyflakes from `PATH` when those integrations are enabled; it does not install th
260260
261261
`--dry-run` enables `--strict`, `--staged`, `--validate`, `--shellcheck`,
262262
and `--models`. These settings are
263-
forced regardless of frontmatter; explicitly disabling a required flag is rejected. Compiler warnings,
264-
including safe-update warnings, and scanner failures cause a nonzero exit status
265-
and invalid JSON validation results. Missing required tools block compilation
263+
forced regardless of frontmatter; explicitly disabling a required flag is rejected.
264+
Enabled scanners first scan the existing lock files before compilation, then scan
265+
the emitted diagnostic locks with the same scanner configuration and result filters.
266+
Only added medium-or-higher findings, or added findings with unknown severity,
267+
fail the scanner gate. Matching uses finding identity and occurrence counts, not
268+
aggregate totals, and tolerates shifted YAML lines and steps with stable IDs/names.
269+
Low-severity and informational findings remain visible but do not fail the gate.
270+
Zizmor's `secrets-outside-env` rule is excluded in dry-run mode because repository
271+
and organization secrets are common in agentic workflows; ordinary compilation
272+
retains the rule. With no existing lock, every actionable finding is added.
273+
The baseline is the on-disk lock at invocation time, not a Git branch comparison;
274+
repeated dry runs compare against the locks left by the preceding invocation.
275+
276+
Validation errors, unclassified compiler warnings (including safe-update warnings),
277+
and scanner execution/parsing failures remain fatal, including failures while
278+
collecting the baseline. Missing required tools block compilation
266279
rather than silently skipping checks. Model checking uses the observed active
267280
model inventory; development compilation fails if that inventory is unavailable
268281
and the workflow declares `models` policies or `engine.models`. It does not prove
@@ -271,12 +284,15 @@ Observed-inventory refresh and collection warnings are also reported: they fail
271284
the dry-run gate even when cached observations remain available. Ordinary
272285
`--models` compilation reports them as warnings instead.
273286
274-
To acknowledge experimental-feature notices explicitly, add `--allow-experimental`
275-
to `--dry-run`. Notices remain visible and the JSON dry-run summary records
276-
`accepted_experimental_warnings`. This accepts only the compiler's built-in
277-
experimental-feature notices, not arbitrary warning text. Security, safe-update,
278-
schedule, model-inventory and other warnings, validation errors, and scanner
279-
failures still block the gate. Required checks and staging remain enabled.
287+
Built-in experimental-feature notices are low-severity and nonblocking.
288+
They remain visible and the JSON summary records `ignored_low_severity_warnings`.
289+
`--allow-experimental` additionally records explicit acknowledgement in
290+
`accepted_experimental_warnings`; it does not exempt arbitrary warning text.
291+
The summary identifies `warning_baseline`, `minimum_severity`, and
292+
`ignored_scanner_rules`; each scanner's `differential` reports baseline, existing,
293+
added, and low-severity ignored counts. A scanner status of `passed` means its
294+
differential gate passed, not that its output contained no findings.
295+
Required checks and staging remain enabled.
280296
281297
Docker-based scanners and `--validate-images` remain opt-in. Docker unavailability
282298
does not block the dry-run gate; install native `shellcheck` for required run-step
@@ -286,7 +302,7 @@ is recommended for source checks, alongside
286302
checks. `validate` uses `--no-emit` and skips these scanners; its success does not
287303
establish scanner coverage. Run the scanner-enabled compile command when possible
288304
and report unavailable checks as unverified. Explicitly requested checks
289-
still fail the dry-run gate on findings or scanner failures.
305+
still fail the dry-run gate on added actionable findings or scanner failures.
290306
291307
When optional CLI scanner preparation cannot find Docker on `PATH`, it prints one
292308
informational notice per process and does not retry image downloads. MCP

‎docs/src/content/docs/setup/cli.md‎

Lines changed: 2 additions & 2 deletions

0 commit comments

Comments
 (0)