fix(agent): recover from a stalled agent bring-up instead of failing the run by slayerjain · Pull Request #4559 · keploy/keploy · GitHub
Skip to content

fix(agent): recover from a stalled agent bring-up instead of failing the run - #4559

Open
slayerjain wants to merge 1 commit into
mainfrom
fix/agent-bringup-retry
Open

fix(agent): recover from a stalled agent bring-up instead of failing the run#4559
slayerjain wants to merge 1 commit into
mainfrom
fix/agent-bringup-retry

Conversation

@slayerjain

Copy link
Copy Markdown
Member

Problem

On docker/macOS CI runners the keploy-agent container intermittently stalls at startup — it produces no logs and never reports healthy — so AgentClient.Setup waits the full agent-ready timeout (330s) and gives up. It's nondeterministic (~37% on the macOS python-docker lanes, and it recurs on main with unchanged code), so one wedged container turns an otherwise-green run red.

Two defects made this both unrecoverable and undiagnosable:

  • Leak: the readiness-timeout branch returned before the if err != nil { stopAgent() } cleanup defer was in scope, so the wedged agent container and startAgent's goroutine leaked (the leaked context is a.agentCancel, which the readiness poll's own context never touches).
  • Black box: nothing captured the agent container's own logs, so a failure was 330s of silence then a generic error — no way to tell a docker run/container-start stall from an in-agent hang.

Fix

  1. On the readiness timeout: dump the agent container's docker logs + state (logAgentContainerDiagnostics, best-effort, 15s-bounded), call stopAgent() to tear the wedged container down (fixes the leak), and return a new pkg.ErrAgentNotReady sentinel.
  2. pkg.RetryAgentSetup retries the bring-up up to 3× only on ErrAgentNotReady. A fresh Setup re-draws ports and regenerates the agent container, which is what clears a nondeterministic runtime stall. The readiness poll happens before any app/proxy is started, so a retry only re-brings-up the agent — the test set still runs once, against a healthy agent. Deterministic failures (missing privileges, bad config) don't match the sentinel and return immediately; a cancelled context stops the loop.
  3. Retries pass a shorter SetupOptions.AgentReadyTimeout (120s): attempt 1 keeps the full slow-start budget (for the legitimately-slow ~126s cold start that 330s exists for), but a fresh agent is ready in ~1–2s, so a wedged retry is cut short. Worst case is bounded at 330+120+120s instead of 3×330s.

This retries the agent infrastructure, never a test assertion — a genuine test failure still fails on the first attempt.

Scope (deliberate)

  • The leak fix + diagnostics live inside Setup, so they apply to every caller (record, replay, keploy mock, runner).
  • The retry wraps the record/replay Instrument bring-up — the path that hits this stall. keploy mock and runner remain single-attempt (they still get the leak fix + diagnostics).
  • The docker-compose per-test-set readiness path is out of scope: Setup skips the readiness poll for compose, so it never returns the sentinel. The observed failure is the plain-docker run poll inside Setup, which this covers. If the stall ever reproduces on a compose lane, those bare-error waits (record.go/replay.go/runner.go) would need the same treatment.

Tests

pkg/retry_agent_setup_test.go verifies: single call on success; retry only on the wrapped sentinel; no retry on a deterministic error; bounded attempts; and immediate stop on context cancellation. go build ./pkg/..., go vet, and the touched-package suites are clean.

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

…the run

On docker/macOS CI runners the keploy-agent container intermittently stalls at
startup — it produces no logs and never reports healthy — so AgentClient.Setup
waits the full agent-ready timeout and gives up. It is nondeterministic
(~37% on the macOS lanes, and it recurs on main with unchanged code), so a
single wedged container turns an otherwise-green run red.

Two defects made this both unrecoverable and undiagnosable:

  * The readiness-timeout branch returned before the `if err != nil` cleanup
    defer was in scope, so the wedged agent container AND its goroutine leaked.
  * Nothing captured the agent container's own logs, so the failure was a black
    box: the whole readiness window of silence, then a generic error, with no
    way to tell a docker-run/container-start stall from an in-agent hang.

Fixes:

  * On the readiness timeout, dump the agent container's `docker logs` and state
    (logAgentContainerDiagnostics), then stopAgent() to tear the wedged
    container down, then return the new pkg.ErrAgentNotReady sentinel.
  * pkg.RetryAgentSetup retries the bring-up up to 3 times, but ONLY on
    ErrAgentNotReady — a fresh Setup re-draws ports and regenerates the agent
    container, which is what clears a nondeterministic runtime stall. The
    readiness poll happens before any app/proxy is started, so a retry only
    re-brings-up the agent; the test set still runs once, against a healthy
    agent. Deterministic failures (missing privileges, bad config) don't match
    the sentinel and return at once; a cancelled context stops the loop.
  * Retries pass a shorter SetupOptions.AgentReadyTimeout (120s): the first
    attempt keeps the full slow-start budget, but a fresh agent reports healthy
    in a second or two, so a wedged retry is cut short. Worst case is bounded at
    330+120+120s rather than 3x330s, keeping it well inside the CI job timeout.

This retries the agent infrastructure, never a test assertion — a genuine test
failure still fails on the first attempt.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
@slayerjain
slayerjain force-pushed the fix/agent-bringup-retry branch from ed1f2a5 to 6d6ca0b Compare September 6, 2026 04:40
@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run P50 P90 P99 RPS Error Rate Status
1 2.7ms 3.42ms 5.03ms 100.00 0.00% ✅ PASS
2 2.75ms 3.5ms 4.86ms 100.02 0.00% ✅ PASS
3 2.69ms 3.47ms 4.92ms 100.02 0.00% ✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant