fix(compose): probe postgres over TCP so doccano isn't released early by slayerjain · Pull Request #106 · keploy/samples-python · GitHub
Skip to content

fix(compose): probe postgres over TCP so doccano isn't released early - #106

Merged
slayerjain merged 1 commit into
mainfrom
fix/pg-healthcheck-tcp
Sep 4, 2026
Merged

fix(compose): probe postgres over TCP so doccano isn't released early#106
slayerjain merged 1 commit into
mainfrom
fix/pg-healthcheck-tcp

Conversation

@slayerjain

Copy link
Copy Markdown
Member

The bug

postgres' docker-entrypoint runs a temporary server reachable only over the Unix socket (listen_addresses='') once initdb has finished, so it can create the database and run docker-entrypoint-initdb.d.

A pg_isready healthcheck without -h talks to that socket. So it reports "accepting connections" while no TCP listener exists at all — compose marks the service healthy, depends_on: condition: service_healthy releases the dependent, and the dependent connects straight into ECONNREFUSED.

It only bites on a fresh volume, because a populated one skips initdb and the temporary server entirely. That is why it surfaces as a rare flake rather than a consistent failure.

Evidence

From the CI failure this was diagnosed from (enterprise umami-linux, a matrix cell that creates its volume):

event time
temp server "ready to accept connections" (socket only) 08:54:29.422 — false-healthy begins
fast shutdown request 08:54:30.668 — window 1.25s
shutdown checkpoint 08:54:30.78008:54:33.421 (2.7s)
real TCP listener after 08:54:33.583

compose logged Container umami_db WaitingContainer umami_app Started, then the app died with Can't reach database server at 172.78.0.10:5432, while the db showed Up 11 seconds (healthy).

Verification

Reproduced and mutation-tested locally with a compose mirroring the depends_on: service_healthy shape:

healthcheck dependent
pg_isready -U ... -d ... APP_FAILED_TO_REACH_DB, exit 1
pg_isready -h 127.0.0.1 -U ... -d ... APP_OK, exit 0

Directly measured the false-healthy window against the patched file: the old check reports healthy from t=2s while TCP is refused until t=8s; the patched check holds starting and only goes healthy once TCP is genuinely up. Reproduced on postgres 13.3, 15 and 16.

Why -h 127.0.0.1

The stock images ship listen_addresses = '*', so the real server binds 0.0.0.0 — one bind() covers loopback and the container IP, and the static ipv4_address is configured before PID 1 runs, so there is no phase where loopback works and the network IP does not. During startup/recovery pg_isready reports rejecting connections, so the probe correctly keeps failing until the server can actually serve. localhost would be the wrong literal (it may resolve ::1 first).

Note on pg_isready semantics

pg_isready is a liveness probe: it exits 0 even for a nonexistent role/database, and even when pg_hba.conf rejects the connection. The -U/-d flags here are effectively decorative. That is fine for this fix — the real listener only starts after initdb created the role and database — but it does mean the probe is not a substitute for an application-level readiness check.

…arly

postgres' docker-entrypoint runs a temporary server reachable only over the
Unix socket (listen_addresses='') once initdb has finished, so it can create
the database and run docker-entrypoint-initdb.d. A pg_isready probe without
-h talks to that socket, so it answers "accepting connections" while no TCP
listener exists at all. compose then marks the service healthy and
depends_on: service_healthy releases the dependent straight into
ECONNREFUSED.

Measured at ~235ms with no initdb.d scripts, and at 1.2s of false-healthy
followed by a 2.7s shutdown checkpoint in the CI failure this was diagnosed
from. Reproduced on postgres 13.3, 15 and 16. Only bites on a fresh volume,
since a populated one skips the temporary server entirely -- which is why it
surfaced as a rare flake rather than a consistent failure.

Passing -h 127.0.0.1 forces a TCP probe, which tracks the real listener in
every phase: the stock images ship listen_addresses = '*', and during startup
or recovery pg_isready reports "rejecting connections", so the probe
correctly keeps failing until the server can actually serve.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

doccano-django sample coverage

ref coverage
base (main) 59.0%
this PR 59.0%

Threshold: PR may not drop coverage by more than 1.0pp. Override per-repo via the DOCCANO_COVERAGE_THRESHOLD actions variable.

@slayerjain
slayerjain merged commit 0823989 into main Sep 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant