A facet's reconnect replay sends one root RPC per stored chunk, and failed sends loop until the isolate runs out of memory · Issue #2606 · cloudflare/agents · GitHub
Skip to content

A facet's reconnect replay sends one root RPC per stored chunk, and failed sends loop until the isolate runs out of memory #2606

Open

Description

@jackdnl

Drafted 2026-10-16 from Joke's production (account schematik, script joke).
Versions: agents 0.28.0, @cloudflare/think 0.20.2, ai 7.0.137, wrangler 4.147.0,
compatibility_date 2026-10-01, nodejs_compat. Object layout: JokeScope (root DO) →
JokeSession (Think facet) → JokeWorker (Think facet, run with runAgentTool, detached).


What happens

A client opens a WebSocket to a Think facet (useAgent({ agent: "joke-scope", sub: [{ agent: "joke-session", ... }] })).
On connect, Agent._replayAgentToolRuns(connection) replays every row in cf_agent_tool_runs, and for each
one every stored chunk of the child (_broadcastAgentToolStoredChunks → _broadcastAgentToolChunks), one
connection.send() per chunk.

In a facet, connection is a virtual bridge connection. send() goes through
DynamicAgents.routeConnectionOperation(id, "send", ...): the first send uses the live frame bridge, but while
it is pending every later send is chained onto the per-connection tail and becomes its own
root._cf_sendToSubAgentConnection(connectionId, message) RPC (RootDynamicAgentConnectionBridge). A failed
operation is only reported:

console.error("[Agent] Sub-agent connection operation failed:", { connectionId, operation, error })

Nothing stops the rest of the replay, drops the connection, backs off, or bounds the number of queued
operations. For a session with long-running agent tools (tens of thousands of stored chunks) a single connect
queues tens of thousands of root RPCs and their messages in memory.

Evidence (Workers Observability, 2026-10-12 UTC)

One Durable Object (da2fb9547a3d8e1ed1ec9fd0146cd4e002f8e3f3ed74a95a9092ad22e9561827, entrypoint JokeSession):

hour (UTC) "[Agent] Sub-agent connection operation failed"
12 9
14 13,889
15 51,532
16 75,180
  • 140,610 failures in total, operation: "send" for all but ~57 (setState), over five connection ids:
    57,223 / 51,516 / 17,925 / 13,889 / 6. One connect replay accounts for up to 57k sends.
  • Each logged error is { "remote": true, "durableObjectId": "<the root's id>" }, i.e. the root RPC failed.
  • The same object logged "Durable Object's isolate exceeded its memory limit and was reset" 78,489 times that
    day (78,489 as JokeSession, 11 as JokeScope), and other objects' connections through that isolate broke.
  • After we bounded the replay ourselves (below), the count went to zero: 0 on each of 10-13, 10-14, 10-15, 10-16.

Repro notes

  1. A Think root with a Think facet that has run a few runAgentTool children which streamed many chunks
    (we had workers with thousands of text/tool chunks each; a scripted child emitting 20k small text deltas is
    enough).
  2. Connect a client to the facet with useAgent({ sub: [...] }) (or a raw socket to the sub-agent URL).
  3. Close the socket from the client side while the replay is running, or connect several clients in a row.
  4. Watch the facet's logs: one operation: "send" failure per remaining chunk, and growing memory.

What would help

  • Bound the replay: the last N chunks per run (clients only need status, latest parts and the result for an old
    run), and only running runs plus the newest N ended ones.
  • Batch a replay into one root call (_cf_sendManyToSubAgentConnection(id, messages[])) instead of one RPC per
    chunk.
  • After a failed operation for a connection, stop sending to that connection (treat it as closed) instead of
    attempting every queued operation; rate-limit the error log.

Our workaround

joke-cloudflare/src/harness/worker-replay.ts wraps _broadcastAgentToolChunks (replay tail of 40 chunks per
run), _broadcastAgentToolEvent and _broadcastAgentToolStoredChunks (replay only running runs and the newest
20 ended ones; older runs' chunks are not even read from the child). We also run our own workers' workers with
eventDelivery: "terminal" because nobody watches their chunks live.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions