{{ message }}
meta: give Db2 containers enough time to initialize before giving up - #18341
Draft
WikiRik wants to merge 2 commits into
Draft
meta: give Db2 containers enough time to initialize before giving up#18341WikiRik wants to merge 2 commits into
WikiRik wants to merge 2 commits into
Conversation
db2 oldest/latest jobs fail regularly in `yarn start-db2-*` with "Timeout of 150s exceeded when waiting for container to be healthy". Db2 creates the database on first start, and successful runs of the same step on main take 127-213s, so the 150s budget of the Db2-only copy of wait-until-healthy.sh is right at the edge of normal. Locally, on a fast machine, the container first reports healthy after 175s. That copy also predates #18291: it still has the dead `return_code` check, prints nothing about the container on failure, and sleeps a blind 15s after the container is healthy. Remove it and use the shared dev/wait-until-healthy.sh, which supports HEALTHCHECK_TIMEOUT and dumps the health probes and container logs when it gives up. Db2 gets a 600s budget so a slow runner has headroom while a genuinely stuck container still fails in minutes rather than hours. The setup script restarts the instance twice after creating the database, so a probe can pass in a short window before a restart. The healthcheck now also requires the image's own setup-complete marker, which is created after the last restart. This replaces the blind 15s sleep. start_period goes to 300s so the container is not reported unhealthy while it is legitimately still creating the database. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
4 tasks
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Pull Request Checklist
yarn testoryarn test-DIALECTpass with this change (including linting)?Description Of Change
Since #18291 landed,
db2 oldestanddb2 latestare the most frequent database start failures in CI: 15 failedyarn start-db2-*steps across 9 runs, all withExamples: db2 oldest (Node 22), db2 latest (Node 24).
Cause
Db2 creates the database on first start. The start step on successful main runs takes 127-213s, and locally on a fast machine the container first reports healthy after 175s. The Db2-only copy of
wait-until-healthy.shpolls for 10 × 15s = 150s, which is right at the edge of a normal startup, so a slightly slow runner fails.That copy also predates #18291: it still has the dead
return_codecheck, prints nothing about the container on failure (which is why the job logs above say nothing useful), and sleeps a blind 15s after the container is healthy.There is a second, latent problem. The image's setup script restarts the instance twice right after
CREATE DATABASE:A probe landing in one of those ~6s windows marks the container healthy, after which the instance goes down again and
check-connection.tsgetsECONNREFUSED. The oldsleep 15was presumably guarding against this.Changes
dev/db2/wait-until-healthy.sh; bothstart.shuse the shareddev/wait-until-healthy.shwithHEALTHCHECK_TIMEOUT=600, so a slow runner has headroom while a genuinely stuck container still fails in minutes rather than hours, and the health probes + container logs are dumped on failure./database/config/.shared-data/setup_complete, the marker the image'ssetup_db2_instance.shtouches right after the last restart (same path in 11.5.5.1 and 12.1.4.0). This replaces the blind sleep.start_period: 300sso the container is not reportedunhealthywhile it is legitimately still creating the database.Verification (local,
icr.io/db2_community/db2:11.5.5.1)Note: touches the same
start.shfiles as #18340, expect a trivial conflict in whichever lands last.🤖 Generated with Claude Code