Don't fail flaky integration check on SKIP_LIST-filtered tests by alexey-milovidov · Pull Request #104542 · ClickHouse/ClickHouse · GitHub
Skip to content

Don't fail flaky integration check on SKIP_LIST-filtered tests#104542

Merged
alexey-milovidov merged 2 commits into
masterfrom
ci-flaky-check-skip-list
May 12, 2026
Merged

Don't fail flaky integration check on SKIP_LIST-filtered tests#104542
alexey-milovidov merged 2 commits into
masterfrom
ci-flaky-check-skip-list

Conversation

@alexey-milovidov

@alexey-milovidov alexey-milovidov commented May 10, 2026

Copy link
Copy Markdown
Member

The flaky integration test job collects changed test modules from the PR diff and asks get_parallel_sequential_tests_to_run to match each one against the parallel/sequential module lists. Without no_strict, a changed test that has been filtered out (e.g. by the private fork's SKIP_LIST in ci/jobs/scripts/integration_tests_configs.py) trips assert matched, f"Test [{test_arg}] not found", killing the whole job.

This is the same concern that already justifies no_strict=True for targeted checks (a CIDB-recorded test may be deleted or renamed before the job runs). Extend the relaxation to flaky checks so a SKIP_LIST'd test is silently dropped, then mirror the targeted-check empty-selection short-circuit so a flaky job whose entire test list was filtered out is reported as SKIPPED instead of falling through to Result.create_from(results=[], ...) and being marked ERROR.

Unblocks CH Inc sync on PRs that modify any test in SKIP_LIST (e.g. test_kafka_bad_messages), such as #100276 — its sync PR https://github.com/ClickHouse/clickhouse-private/pull/53191 failed in Integration tests (amd_asan_ubsan, flaky) with AssertionError: Test [test_kafka_bad_messages/test.py] not found.

Changelog category (leave one):

  • CI Fix or Improvement (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

...

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

Version info

  • Merged into: 26.5.1.563

The flaky integration test job collects changed test modules from the PR
diff and asks `get_parallel_sequential_tests_to_run` to match each one
against the parallel/sequential module lists. Without `no_strict`, a
changed test that has been filtered out (e.g. by the private fork's
`SKIP_LIST` in `ci/jobs/scripts/integration_tests_configs.py`) trips
`assert matched, f"Test [{test_arg}] not found"`, killing the whole job.

This is the same concern that already justifies `no_strict=True` for
targeted checks (a CIDB-recorded test may be deleted or renamed before
the job runs). Extend the relaxation to flaky checks so a SKIP_LIST'd
test is silently dropped — the private fork has an explicit empty-result
handler that turns the resulting empty test list into a `SKIPPED` job.

This unblocks `CH Inc sync` on PRs that modify any test in `SKIP_LIST`
(e.g. `test_kafka_bad_messages`), such as
#100276 — its sync PR
ClickHouse/clickhouse-private#53191 failed in
`Integration tests (amd_asan_ubsan, flaky)` with
`AssertionError: Test [test_kafka_bad_messages/test.py] not found`.
@clickhouse-gh

clickhouse-gh Bot commented May 10, 2026

Copy link
Copy Markdown
Contributor

@clickhouse-gh clickhouse-gh Bot added the pr-ci label May 10, 2026
Comment thread ci/jobs/integration_test_job.py
The prior commit relaxed `get_parallel_sequential_tests_to_run` with
`no_strict=True` for flaky checks so a `SKIP_LIST`-filtered test would no
longer trip the `Test [...] not found` assertion. But the empty-selection
path was only handled for targeted checks: if both module lists came out
empty for a flaky check, no `pytest` run happened and
`Result.create_from(results=[], ...)` ended the job with `ERROR` rather
than `SKIPPED`.

Extend the empty-selection short-circuit to flaky checks so the job is
reported as `SKIPPED` with an explanatory `info` message.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@alexey-milovidov alexey-milovidov requested a review from maxknv May 12, 2026 15:23
@alexey-milovidov alexey-milovidov self-assigned this May 12, 2026
@alexey-milovidov alexey-milovidov added this pull request to the merge queue May 12, 2026
Merged via the queue into master with commit fe85771 May 12, 2026
165 checks passed
@alexey-milovidov alexey-milovidov deleted the ci-flaky-check-skip-list branch May 12, 2026 15:36
@robot-ch-test-poll1 robot-ch-test-poll1 added the pr-synced-to-cloud The PR is synced to the cloud repo label May 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-ci pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants