Makefile: run every benchmark platform and report failures to Buildkite by copybara-service[bot] · Pull Request #16079 · google/gvisor · GitHub
Skip to content

Makefile: run every benchmark platform and report failures to Buildkite - #16079

Merged
copybara-service[bot] merged 2 commits into
masterfrom
test/cl1003444561
Oct 21, 2026
Merged

copybara-service[bot] merged 2 commits into
masterfrom
test/cl1003444561

Conversation

@copybara-service

Copy link
Copy Markdown

Makefile: run every benchmark platform and report failures to Buildkite

benchmark-platforms runs the platforms in a set -e loop and does not capture each platform's exit status, so the first platform that fails aborts the loop and the remaining platforms never run. Today kvm fails on every run (BenchmarkRestore* is broken on Docker 28, moby/moby#50750), so systrap never runs and systrap data is never populated.

The failure also never surfaced in Buildkite. The benchmark steps run make -i, which ignores the error and exits 0, so Buildkite marked the step as passed and the step's soft_fail: true was never used.

Changes:

  • benchmark-platforms, benchmark-platforms-nodocker: capture each platform's exit status (|| exit_code=$?), run all platforms and runc, then exit $exit_code.
  • run_platform_benchmark: on failure, still upload the parsed results and remove the log file, and return the benchmark's exit status (same structure as run_benchmark).
  • .buildkite/pipeline.yaml: remove -i from the benchmark steps. The Makefile now continues past failures.

Effect: every platform runs and uploads its passing benchmarks even if an earlier platform failed. A step with a failing benchmark exits non-zero, which soft_fail: true turns into a yellow soft-failed step; the build still passes. Until BenchmarkRestore* is fixed, the Startup step will be yellow on every run.
FUTURE_COPYBARA_INTEGRATE_REVIEW=#16078 from riverafernando:fix-benchmark-platforms 79ed273

`benchmark-platforms` runs the platforms in a `set -e` loop and does not
capture each platform's exit status, so the first platform that fails
aborts the loop and the remaining platforms never run. Today `kvm` fails on
every run (`BenchmarkRestore*` is broken on Docker 28,
moby/moby#50750), so `systrap` never runs and the
Startup dashboard has no `systrap` data.

The failure also never surfaced in Buildkite. The benchmark steps run
`make -i`, which ignores the error and exits 0, so Buildkite marked the
step as passed and the step's `soft_fail: true` was never used.

Changes:
- `benchmark-platforms`, `benchmark-platforms-nodocker`: capture each
  platform's exit status (`|| exit_code=$?`), run all platforms and `runc`,
  then `exit $exit_code`.
- `run_platform_benchmark`: on failure, still upload the parsed results and
  remove the log file, and return the benchmark's exit status (same
  structure as `run_benchmark`).
- `.buildkite/pipeline.yaml`: remove `-i` from the benchmark steps. The
  Makefile now continues past failures.

Effect: every platform runs and uploads its passing benchmarks even if an
earlier platform failed. A step with a failing benchmark exits non-zero,
which `soft_fail: true` turns into a yellow soft-failed step; the build
still passes. Until `BenchmarkRestore*` is fixed, the Startup step will be
yellow on every run.
@copybara-service copybara-service Bot added the exported Issue was exported automatically label Oct 20, 2026
@copybara-service
copybara-service Bot force-pushed the test/cl1003444561 branch 2 times, most recently from 02a1911 to 46948bd Compare October 21, 2026 17:10
@copybara-service
copybara-service Bot merged commit 325ed2e into master Oct 21, 2026
7 of 9 checks passed
@copybara-service
copybara-service Bot deleted the test/cl1003444561 branch October 21, 2026 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

exported Issue was exported automatically

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants