iframe-proxy

groeneai · 2026-06-27T01:20:32Z

Changelog category (leave one):

CI Fix or Improvement (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

...

Description

The Docker keeper image job intermittently fails while fetching the prebuilt packages from clickhouse-builds.s3.amazonaws.com. The clickhouse-common-static package downloads fine, then the clickhouse-keeper package gets a 503 Service Unavailable and the single-shot wget ... || exit 1 kills the whole build. The downstream docker library image test then reports image does not exist! (a secondary victim, not a second failure).

Recent example, PR #108573 (docker/keeper/Dockerfile.distroless, amd64):

--2026-06-26 22:05:41--  https://clickhouse-builds.s3.amazonaws.com/PRs/108573/.../build_amd_release/clickhouse-keeper_26.7.1.1_amd64.deb
HTTP request sent, awaiting response... 503 Service Unavailable
2026-06-26 22:05:41 ERROR 503: Service Unavailable.
Dockerfile.distroless:43
ERROR: failed to build: ... exit code: 1

The server Dockerfiles already wrap wget with a retry helper for exactly this transient-S3 class: docker/server/Dockerfile.alpine since #100380 (tgz path) and docker/server/Dockerfile.ubuntu / docker/server/Dockerfile.distroless since #105139 (deb path). The Keeper Dockerfiles were left calling wget on the first attempt only, so a single 500/503 from S3 fails the build.

This ports the same wget_with_retry wrapper to both Keeper Dockerfiles:

docker/keeper/Dockerfile.distroless: the deb DIRECT_DOWNLOAD_URLS path (the one CI uses), mirroring docker/server/Dockerfile.distroless.
docker/keeper/Dockerfile (alpine/ubuntu): the tgz DIRECT_DOWNLOAD_URLS path and the repository fallback, mirroring docker/server/Dockerfile.alpine.

WGET_RETRIES (default 5) and WGET_RETRY_DELAY (default 1s) match the server Dockerfiles and remain overridable via --build-arg. A persistent error still fails the build after the retries are exhausted, so a genuinely-broken S3 is not masked.

Recurrence (30 days): Docker keeper image FAIL = 165 hits across 15 distinct PRs + 21 on master, all the scattered transient-S3 signature that re-runs clean.

CI report (PR #108573): https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=108573&sha=1766fdb7f5dd9384c444db648ceaa4a4548dc4db&name_0=PR&name_1=Docker%20keeper%20image

Version info

Merged into: 26.7.1.196

The `Docker keeper image` job intermittently fails while fetching the prebuilt packages from `clickhouse-builds.s3.amazonaws.com`. Example from PR ClickHouse#108573 (`docker/keeper/Dockerfile.distroless`, amd64): the `clickhouse-common-static` deb downloads fine, then the `clickhouse-keeper` deb gets HTTP request sent, awaiting response... 503 Service Unavailable ERROR 503: Service Unavailable. and the single-shot `wget ... || exit 1` kills the whole build, so the downstream `docker library image test` then reports `image does not exist!`. The server Dockerfiles already wrap `wget` with a retry helper: `Dockerfile.alpine` since ClickHouse#100380 (tgz path) and `Dockerfile.ubuntu` / `Dockerfile.distroless` since ClickHouse#105139 (deb path). The Keeper Dockerfiles were left with the brittle first-attempt `wget`, so a single 500/503 from S3 fails the build. This ports the same `wget_with_retry` wrapper to both Keeper files: - `docker/keeper/Dockerfile.distroless` deb path (`DIRECT_DOWNLOAD_URLS`), mirroring `docker/server/Dockerfile.distroless`. - `docker/keeper/Dockerfile` tgz path (`DIRECT_DOWNLOAD_URLS` and the repository fallback), mirroring `docker/server/Dockerfile.alpine`. `WGET_RETRIES` (default 5) and `WGET_RETRY_DELAY` (default 1s) match the server Dockerfiles and stay overridable via `--build-arg`. A persistent error still fails the build after the retries are exhausted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

groeneai · 2026-06-27T01:20:51Z

Pre-PR validation gate (click to expand)

#	Question	Answer
a	Deterministic repro?	Yes. Build log on PR #108573 shows `clickhouse-keeper_26.7.1.1_amd64.deb` -> `503 Service Unavailable` -> `Dockerfile.distroless:43` `wget ... \|\| exit 1` fails the build. Reproduced the RUN-block behavior locally with a mock `wget` returning 503 N times: OLD code exits 1 on the first 503; NEW code retries and succeeds.
b	Root cause explained?	The CI download path (`DIRECT_DOWNLOAD_URLS`) in both Keeper Dockerfiles calls `wget` once and `\|\| exit 1` on the first failure. `clickhouse-builds.s3.amazonaws.com` intermittently returns `500`/`503`, so a single transient error kills the whole image build; the downstream `docker library image test` then reports `image does not exist!`.
c	Fix matches root cause?	Yes. Wraps the fetch in `wget_with_retry` (5 attempts, 1s delay) so a transient `5xx` is retried instead of aborting. This is the exact mechanism that addressed the same failure on the server Dockerfiles (#100380, #105139).
d	Test intent preserved / new tests added?	N/A (Docker/CI-infra change, no test suite). Persistent errors still fail the build after retries are exhausted, so a genuinely-broken S3 is not masked.
e	Both directions demonstrated?	Yes. With a mock `wget`: transient 503x2-then-success -> OLD exits 1 (build fails), NEW exits 0 (build proceeds); persistent 503 -> NEW still exits 1. Validated under both `dash` (distroless/ubuntu builder shell) and `busybox sh` (alpine shell).
f	Fix is general across code paths?	Yes. Fixed the class, not one site: `Dockerfile.distroless` (deb) AND `Dockerfile` (tgz: both the `DIRECT_DOWNLOAD_URLS` loop and the repository fallback). The server siblings already had this; this completes Keeper. The single-binary path is not used for Keeper CI builds and is left unchanged to keep the diff minimal (matches the server #105139 scope).
g	Fix generalizes across inputs?	N/A (CI infra change, not a code bug over typed inputs). The retry wraps any URL/arch passed by CI.
h	Backward compatible?	Yes. `WGET_RETRIES`/`WGET_RETRY_DELAY` are new `ARG`s with the same defaults as the server Dockerfiles, overridable via `--build-arg`. No image contents, layers, or behavior change on success.
i	Invariants and contracts preserved?	Yes. Same `set -e`/`&&`-chain semantics; the helper returns non-zero on exhaustion so the `\|\| exit 1` and the `&&` chain still abort on real failures. sha512 verification and unpack steps are unchanged.

Session id: cron:clickhouse-worker-slot-5:20260627-010600

groeneai · 2026-06-27T01:21:24Z

cc @Algunenano — could you review this? It ports the wget_with_retry wrapper (already in the server Dockerfiles via #100380 and #105139) to the two Keeper Dockerfiles, so a transient S3 500/503 while fetching the keeper .deb/.tgz retries instead of failing the Docker keeper image build on the first attempt.

clickhouse-gh · 2026-06-27T16:38:10Z

Workflow [PR], commit [6a2d2ce]

Summary: ✅

AI Review

Summary

This PR ports the existing server Dockerfile wget_with_retry pattern to the Keeper image download paths that CI exercises. The changed DIRECT_DOWNLOAD_URLS and repository package fetches preserve the previous failure behavior after retry exhaustion while covering transient 500/503 download failures. I found no correctness, safety, or PR metadata issues that need inline comments.

Final Verdict

Status: ✅ Approve

clickhouse-gh · 2026-06-28T06:37:16Z

📊 Cloud Performance Report

✅ AI verdict: no_change — no significant changes across 38 queries analysed

This PR only adds wget retry logic to the keeper Dockerfiles (both private and public diffs are identical and Docker-build-only). Nothing in it touches the server's query-execution path, so the flagged improvements on ClickBench Q4 (-13.88%) and Q15 (-14.35%) cannot be caused by this change. Both are downgraded to not-sure as run-to-run measurement variance. No action needed on these queries.

clickbench

⚠️ 2 inconclusive

Flagged queries (2 of 43)

_{q-value = BH-FDR adjusted p; smaller is stronger evidence. MIRAI flags a query when q < fdr_q (default 0.10) — the value the verdict is based on.}

tpch_adapted_1_official

🟢 No significant changes

Debug info

StressHouse run: 26b44bbd-0e69-4669-ada5-b47e5d212af5
MIRAI run: 9a54702f-535c-4c96-b097-1f06edbc7daf
PR check IDs:
- clickbench_286902_1782627527
- clickbench_286908_1782627527
- clickbench_286918_1782627527
- tpch_adapted_1_official_286926_1782627527
- tpch_adapted_1_official_286937_1782627527
- tpch_adapted_1_official_286947_1782627527

The "Docker server image" job builds the distroless/alpine/ubuntu images with `docker buildx build`. buildx resolves the SBOM scanner image (docker/buildkit-syft-scanner, pulled because of --sbom=true) and base images from docker.io. When docker.io has a transient outage, the build fails at: #2 resolve image config for docker-image://docker.io/docker/buildkit-syft-scanner:stable-1 #2 ERROR: unexpected status from HEAD request to https://registry-1.docker.io/... and the downstream "docker library image test" then reports "image does not exist!" as a secondary victim. praktika's Shell.run / Result.from_commands_run already support retries with exponential backoff and an error allowlist, but the buildx build and the imagetools merge commands ran with no retries. Wire retries=5 and an allowlist of transient registry/network signatures into both. The allowlist keeps fail-fast behavior for genuine Dockerfile/build errors. This mirrors the existing retry hardening for the in-Dockerfile wget downloads (ClickHouse#105139) and the keeper image S3 artifact fetch (ClickHouse#108675). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

groeneai mentioned this pull request Jun 27, 2026

Upload S3 file parts from an in-memory copy to make retries safe #108573

Open

alexey-milovidov added the can be tested Allows running workflows for external contributors label Jun 27, 2026

clickhouse-gh Bot added the pr-ci label Jun 27, 2026

alexey-milovidov approved these changes Jun 28, 2026

View reviewed changes

alexey-milovidov self-assigned this Jun 28, 2026

alexey-milovidov added this pull request to the merge queue Jun 28, 2026

Merged via the queue into ClickHouse:master with commit f1abdda Jun 28, 2026
169 checks passed

robot-clickhouse-ci-2 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jun 28, 2026

groeneai mentioned this pull request Jun 30, 2026

Retry docker buildx on transient docker.io registry errors #108969

Merged

Sunbelt Computer Software

PL/B Language Development and Support

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Retry wget downloads in Keeper Dockerfiles#108675

Retry wget downloads in Keeper Dockerfiles#108675
alexey-milovidov merged 1 commit into
ClickHouse:masterfrom
groeneai:fix-keeper-docker-wget-retry

groeneai commented Jun 27, 2026 •

edited by robot-ch-test-poll

Loading

Uh oh!

groeneai commented Jun 27, 2026

Uh oh!

groeneai commented Jun 27, 2026

Uh oh!

clickhouse-gh Bot commented Jun 27, 2026 •

edited

Loading

Uh oh!

clickhouse-gh Bot commented Jun 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Sunbelt Computer Software

PL/B Language Development and Support

Uh oh!

Conversation

groeneai commented Jun 27, 2026 • edited by robot-ch-test-poll Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Description

Version info

Uh oh!

groeneai commented Jun 27, 2026

Uh oh!

groeneai commented Jun 27, 2026

Uh oh!

clickhouse-gh Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AI Review

Summary

Final Verdict

Uh oh!

clickhouse-gh Bot commented Jun 28, 2026

clickbench

tpch_adapted_1_official

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

groeneai commented Jun 27, 2026 •

edited by robot-ch-test-poll

Loading

clickhouse-gh Bot commented Jun 27, 2026 •

edited

Loading