{{ message }}
fix(agent): recover from a stalled agent bring-up instead of failing the run - #4559
Open
slayerjain wants to merge 1 commit into
Open
fix(agent): recover from a stalled agent bring-up instead of failing the run#4559slayerjain wants to merge 1 commit into
slayerjain wants to merge 1 commit into
Conversation
…the run
On docker/macOS CI runners the keploy-agent container intermittently stalls at
startup — it produces no logs and never reports healthy — so AgentClient.Setup
waits the full agent-ready timeout and gives up. It is nondeterministic
(~37% on the macOS lanes, and it recurs on main with unchanged code), so a
single wedged container turns an otherwise-green run red.
Two defects made this both unrecoverable and undiagnosable:
* The readiness-timeout branch returned before the `if err != nil` cleanup
defer was in scope, so the wedged agent container AND its goroutine leaked.
* Nothing captured the agent container's own logs, so the failure was a black
box: the whole readiness window of silence, then a generic error, with no
way to tell a docker-run/container-start stall from an in-agent hang.
Fixes:
* On the readiness timeout, dump the agent container's `docker logs` and state
(logAgentContainerDiagnostics), then stopAgent() to tear the wedged
container down, then return the new pkg.ErrAgentNotReady sentinel.
* pkg.RetryAgentSetup retries the bring-up up to 3 times, but ONLY on
ErrAgentNotReady — a fresh Setup re-draws ports and regenerates the agent
container, which is what clears a nondeterministic runtime stall. The
readiness poll happens before any app/proxy is started, so a retry only
re-brings-up the agent; the test set still runs once, against a healthy
agent. Deterministic failures (missing privileges, bad config) don't match
the sentinel and return at once; a cancelled context stops the loop.
* Retries pass a shorter SetupOptions.AgentReadyTimeout (120s): the first
attempt keeps the full slow-start budget, but a fresh agent reports healthy
in a second or two, so a wedged retry is cut short. Worst case is bounded at
330+120+120s rather than 3x330s, keeping it well inside the CI job timeout.
This retries the agent infrastructure, never a test assertion — a genuine test
failure still fails on the first attempt.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
slayerjain
force-pushed
the
fix/agent-bringup-retry
branch
from
September 6, 2026 04:40
ed1f2a5 to
6d6ca0b
Compare
🚀 Keploy Performance Test ResultsMulti-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression. Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1% ✅ Result: PASSED - Only 0 out of 3 runs failed (threshold: 2) P50, P90, and P99 percentiles naturally filter out outliers |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Problem
On docker/macOS CI runners the keploy-agent container intermittently stalls at startup — it produces no logs and never reports healthy — so
AgentClient.Setupwaits the full agent-ready timeout (330s) and gives up. It's nondeterministic (~37% on the macOS python-docker lanes, and it recurs onmainwith unchanged code), so one wedged container turns an otherwise-green run red.Two defects made this both unrecoverable and undiagnosable:
if err != nil { stopAgent() }cleanup defer was in scope, so the wedged agent container andstartAgent's goroutine leaked (the leaked context isa.agentCancel, which the readiness poll's own context never touches).docker run/container-start stall from an in-agent hang.Fix
docker logs+ state (logAgentContainerDiagnostics, best-effort, 15s-bounded), callstopAgent()to tear the wedged container down (fixes the leak), and return a newpkg.ErrAgentNotReadysentinel.pkg.RetryAgentSetupretries the bring-up up to 3× only onErrAgentNotReady. A freshSetupre-draws ports and regenerates the agent container, which is what clears a nondeterministic runtime stall. The readiness poll happens before any app/proxy is started, so a retry only re-brings-up the agent — the test set still runs once, against a healthy agent. Deterministic failures (missing privileges, bad config) don't match the sentinel and return immediately; a cancelled context stops the loop.SetupOptions.AgentReadyTimeout(120s): attempt 1 keeps the full slow-start budget (for the legitimately-slow ~126s cold start that 330s exists for), but a fresh agent is ready in ~1–2s, so a wedged retry is cut short. Worst case is bounded at 330+120+120s instead of 3×330s.This retries the agent infrastructure, never a test assertion — a genuine test failure still fails on the first attempt.
Scope (deliberate)
Setup, so they apply to every caller (record, replay,keploy mock, runner).Instrumentbring-up — the path that hits this stall.keploy mockandrunnerremain single-attempt (they still get the leak fix + diagnostics).Setupskips the readiness poll for compose, so it never returns the sentinel. The observed failure is the plain-docker runpoll insideSetup, which this covers. If the stall ever reproduces on a compose lane, those bare-error waits (record.go/replay.go/runner.go) would need the same treatment.Tests
pkg/retry_agent_setup_test.goverifies: single call on success; retry only on the wrapped sentinel; no retry on a deterministic error; bounded attempts; and immediate stop on context cancellation.go build ./pkg/...,go vet, and the touched-package suites are clean.