{{ message }}
Repository navigation
Makefile: run every benchmark platform and report failures to Buildkite - #16079
Merged
Merged
Conversation
`benchmark-platforms` runs the platforms in a `set -e` loop and does not capture each platform's exit status, so the first platform that fails aborts the loop and the remaining platforms never run. Today `kvm` fails on every run (`BenchmarkRestore*` is broken on Docker 28, moby/moby#50750), so `systrap` never runs and the Startup dashboard has no `systrap` data. The failure also never surfaced in Buildkite. The benchmark steps run `make -i`, which ignores the error and exits 0, so Buildkite marked the step as passed and the step's `soft_fail: true` was never used. Changes: - `benchmark-platforms`, `benchmark-platforms-nodocker`: capture each platform's exit status (`|| exit_code=$?`), run all platforms and `runc`, then `exit $exit_code`. - `run_platform_benchmark`: on failure, still upload the parsed results and remove the log file, and return the benchmark's exit status (same structure as `run_benchmark`). - `.buildkite/pipeline.yaml`: remove `-i` from the benchmark steps. The Makefile now continues past failures. Effect: every platform runs and uploads its passing benchmarks even if an earlier platform failed. A step with a failing benchmark exits non-zero, which `soft_fail: true` turns into a yellow soft-failed step; the build still passes. Until `BenchmarkRestore*` is fixed, the Startup step will be yellow on every run.
copybara-service
Bot
requested review from
EtiennePerot and
relkochta
as code owners
October 20, 2026 22:43
copybara-service
Bot
force-pushed
the
test/cl1003444561
branch
2 times, most recently
from
October 21, 2026 17:10
02a1911 to
46948bd
Compare
PiperOrigin-RevId: 1004034061
copybara-service
Bot
force-pushed
the
test/cl1003444561
branch
from
October 21, 2026 17:17
46948bd to
325ed2e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Makefile: run every benchmark platform and report failures to Buildkite
benchmark-platformsruns the platforms in aset -eloop and does not capture each platform's exit status, so the first platform that fails aborts the loop and the remaining platforms never run. Todaykvmfails on every run (BenchmarkRestore*is broken on Docker 28, moby/moby#50750), sosystrapnever runs andsystrapdata is never populated.The failure also never surfaced in Buildkite. The benchmark steps run
make -i, which ignores the error and exits 0, so Buildkite marked the step as passed and the step'ssoft_fail: truewas never used.Changes:
benchmark-platforms,benchmark-platforms-nodocker: capture each platform's exit status (|| exit_code=$?), run all platforms andrunc, thenexit $exit_code.run_platform_benchmark: on failure, still upload the parsed results and remove the log file, and return the benchmark's exit status (same structure asrun_benchmark)..buildkite/pipeline.yaml: remove-ifrom the benchmark steps. The Makefile now continues past failures.Effect: every platform runs and uploads its passing benchmarks even if an earlier platform failed. A step with a failing benchmark exits non-zero, which
soft_fail: trueturns into a yellow soft-failed step; the build still passes. UntilBenchmarkRestore*is fixed, the Startup step will be yellow on every run.FUTURE_COPYBARA_INTEGRATE_REVIEW=#16078 from riverafernando:fix-benchmark-platforms 79ed273