{{ message }}
Report fuzzer runner aborts instead of an opaque status.tsv FileNotFoundError#107676
Merged
alexey-milovidov merged 3 commits intoJun 22, 2026
Conversation
…undError When the AST/BuzzHouse fuzzer process is killed before it writes status.tsv (job timeout, OOM, or a docker/orchestration failure), run_fuzz_job used to fail with an opaque "Unknown error in fuzzer runner script" FileNotFoundError traceback. That gives no hint of the real abort cause and reads like a code crash, even though it is a CI-infrastructure flake that a re-run clears. Distinguish a missing/empty status.tsv (runner aborted early -> infra) from a malformed status.tsv (a real harness bug) and, for the former, emit an actionable message that says the runner aborted before reporting and inlines the tail of the fuzzer/server/stderr logs so the abort cause is visible. The malformed case keeps the traceback. Status stays ERROR so the check is still red and a maintainer re-runs it. The parsing and message formatting are extracted into dependency-free helpers (_read_fuzzer_status, _format_status_error, _log_tail) and covered by unit tests in ci/tests/test_e2e.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
Contributor
Author
|
cc @george-larionov — could you review this? It makes |
nikitamikhaylov
added
the
can be tested
Contributor
![clickhouse-gh[bot]](https://avatars.githubusercontent.com/in/928874?s=60&v=4){kind=small}
A missing status.tsv does not uniquely mean CI infrastructure. run-fuzzer.sh runs under 'set -e' and writes status.tsv only at the very end, so any earlier failure (a server startup failure where the pid file is never created, a fuzzer-harness error, or an infra timeout/OOM/docker problem) also reaches the read site. Asserting infra and 'a re-run usually clears it' would hide a real server-startup or harness regression. Reword the missing-status branch to list the possible causes neutrally and direct the reader to the inlined log tails to determine the actual cause, without claiming infra or suggesting a re-run. The malformed-status branch is unchanged. Rename the unit test to assert the neutral shape (no infra claim, no 're-run'). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run-fuzzer.sh writes only the three numeric fields (server_died, server_exit_code, fuzzer_exit_code) to status.tsv. The FAIL verdict and stack trace are produced later by run_fuzz_job / FuzzerLogParser after a complete status file is parsed. The diagnostic said a finding "writes status.tsv with a FAIL status and a stack trace", which mis-described the contract; reword it to say a finding writes a complete status.tsv that is then reported as FAIL from the logs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 17, 2026
robot-ch-test-poll
added
the
pr-synced-to-cloud
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
...
Description
When the AST/BuzzHouse fuzzer process is killed before it writes
status.tsv(job timeout, OOM, or a docker/orchestration failure),run_fuzz_jobreported an opaqueUnknown error in fuzzer runner scriptfollowed by aFileNotFoundErrortraceback. That gives no hint of the real abort cause and reads like a code crash, even though it is a CI-infrastructure flake that a re-run clears. Over the last 7 days this surfaced as spurious redBuzzHouse (*)/AST fuzzer (*)checks across ~18 unrelated PRs (and master), typically with only one of the four BuzzHouse builds failing on a given SHA while the other three pass on the identical commit.This change distinguishes the two cases:
status.tsv(the runner aborted before reporting): emit an actionable message that says the runner aborted before producing status, explains it is a CI-infrastructure issue rather than a fuzzer finding (a real finding writesstatus.tsvwith aFAILstatus and a stack trace), and inlines the tail of thefuzzer.log/server.log/stderr.logartifacts so the underlying abort cause is visible.status.tsv(a genuine harness bug): keep the traceback for debugging.The status stays
ERRORso the check is still red and a maintainer re-runs it; no auto-retry signal is invented (praktika has no retry path for python-callable jobs).The parsing and message formatting are extracted into dependency-free helpers (
_read_fuzzer_status,_format_status_error,_log_tail) and covered by unit tests inci/tests/test_e2e.py(valid / missing / empty / malformed status, and the infra-vs-harness message shape).No related open issue found (searched the
status.tsv/ast_fuzzer_jobsignature on 2026-06-16).