{{ message }}
Don't fail flaky integration check on SKIP_LIST-filtered tests#104542
Merged
Conversation
The flaky integration test job collects changed test modules from the PR
diff and asks `get_parallel_sequential_tests_to_run` to match each one
against the parallel/sequential module lists. Without `no_strict`, a
changed test that has been filtered out (e.g. by the private fork's
`SKIP_LIST` in `ci/jobs/scripts/integration_tests_configs.py`) trips
`assert matched, f"Test [{test_arg}] not found"`, killing the whole job.
This is the same concern that already justifies `no_strict=True` for
targeted checks (a CIDB-recorded test may be deleted or renamed before
the job runs). Extend the relaxation to flaky checks so a SKIP_LIST'd
test is silently dropped — the private fork has an explicit empty-result
handler that turns the resulting empty test list into a `SKIPPED` job.
This unblocks `CH Inc sync` on PRs that modify any test in `SKIP_LIST`
(e.g. `test_kafka_bad_messages`), such as
#100276 — its sync PR
ClickHouse/clickhouse-private#53191 failed in
`Integration tests (amd_asan_ubsan, flaky)` with
`AssertionError: Test [test_kafka_bad_messages/test.py] not found`.
1 task
Contributor
The prior commit relaxed `get_parallel_sequential_tests_to_run` with `no_strict=True` for flaky checks so a `SKIP_LIST`-filtered test would no longer trip the `Test [...] not found` assertion. But the empty-selection path was only handled for targeted checks: if both module lists came out empty for a flaky check, no `pytest` run happened and `Result.create_from(results=[], ...)` ended the job with `ERROR` rather than `SKIPPED`. Extend the empty-selection short-circuit to flaky checks so the job is reported as `SKIPPED` with an explanatory `info` message. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

The flaky integration test job collects changed test modules from the PR diff and asks
get_parallel_sequential_tests_to_runto match each one against the parallel/sequential module lists. Withoutno_strict, a changed test that has been filtered out (e.g. by the private fork'sSKIP_LISTinci/jobs/scripts/integration_tests_configs.py) tripsassert matched, f"Test [{test_arg}] not found", killing the whole job.This is the same concern that already justifies
no_strict=Truefor targeted checks (a CIDB-recorded test may be deleted or renamed before the job runs). Extend the relaxation to flaky checks so aSKIP_LIST'd test is silently dropped, then mirror the targeted-check empty-selection short-circuit so a flaky job whose entire test list was filtered out is reported asSKIPPEDinstead of falling through toResult.create_from(results=[], ...)and being markedERROR.Unblocks
CH Inc syncon PRs that modify any test inSKIP_LIST(e.g.test_kafka_bad_messages), such as #100276 — its sync PR https://github.com/ClickHouse/clickhouse-private/pull/53191 failed inIntegration tests (amd_asan_ubsan, flaky)withAssertionError: Test [test_kafka_bad_messages/test.py] not found.Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
...
Documentation entry for user-facing changes
Version info
26.5.1.563