A facet's reconnect replay sends one root RPC per stored chunk, and failed sends loop until the isolate runs out of memory · Issue #2606 · cloudflare/agents · GitHub
You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
A facet's reconnect replay sends one root RPC per stored chunk, and failed sends loop until the isolate runs out of memory #2606
Drafted 2026-10-16 from Joke's production (account schematik, script joke).
Versions: agents 0.28.0, @cloudflare/think 0.20.2, ai 7.0.137, wrangler 4.147.0,
compatibility_date 2026-10-01, nodejs_compat. Object layout: JokeScope (root DO) → JokeSession (Think facet) → JokeWorker (Think facet, run with runAgentTool, detached).
What happens
A client opens a WebSocket to a Think facet (useAgent({ agent: "joke-scope", sub: [{ agent: "joke-session", ... }] })).
On connect, Agent._replayAgentToolRuns(connection) replays every row in cf_agent_tool_runs, and for each
one every stored chunk of the child (_broadcastAgentToolStoredChunks → _broadcastAgentToolChunks), one connection.send() per chunk.
In a facet, connection is a virtual bridge connection. send() goes through DynamicAgents.routeConnectionOperation(id, "send", ...): the first send uses the live frame bridge, but while
it is pending every later send is chained onto the per-connection tail and becomes its own root._cf_sendToSubAgentConnection(connectionId, message) RPC (RootDynamicAgentConnectionBridge). A failed
operation is only reported:
Nothing stops the rest of the replay, drops the connection, backs off, or bounds the number of queued
operations. For a session with long-running agent tools (tens of thousands of stored chunks) a single connect
queues tens of thousands of root RPCs and their messages in memory.
Evidence (Workers Observability, 2026-10-12 UTC)
One Durable Object (da2fb9547a3d8e1ed1ec9fd0146cd4e002f8e3f3ed74a95a9092ad22e9561827, entrypoint JokeSession):
hour (UTC)
"[Agent] Sub-agent connection operation failed"
12
9
14
13,889
15
51,532
16
75,180
140,610 failures in total, operation: "send" for all but ~57 (setState), over five connection ids:
57,223 / 51,516 / 17,925 / 13,889 / 6. One connect replay accounts for up to 57k sends.
Each logged error is { "remote": true, "durableObjectId": "<the root's id>" }, i.e. the root RPC failed.
The same object logged "Durable Object's isolate exceeded its memory limit and was reset" 78,489 times that
day (78,489 as JokeSession, 11 as JokeScope), and other objects' connections through that isolate broke.
After we bounded the replay ourselves (below), the count went to zero: 0 on each of 10-13, 10-14, 10-15, 10-16.
Repro notes
A Think root with a Think facet that has run a few runAgentTool children which streamed many chunks
(we had workers with thousands of text/tool chunks each; a scripted child emitting 20k small text deltas is
enough).
Connect a client to the facet with useAgent({ sub: [...] }) (or a raw socket to the sub-agent URL).
Close the socket from the client side while the replay is running, or connect several clients in a row.
Watch the facet's logs: one operation: "send" failure per remaining chunk, and growing memory.
What would help
Bound the replay: the last N chunks per run (clients only need status, latest parts and the result for an old
run), and only running runs plus the newest N ended ones.
Batch a replay into one root call (_cf_sendManyToSubAgentConnection(id, messages[])) instead of one RPC per
chunk.
After a failed operation for a connection, stop sending to that connection (treat it as closed) instead of
attempting every queued operation; rate-limit the error log.
Our workaround
joke-cloudflare/src/harness/worker-replay.ts wraps _broadcastAgentToolChunks (replay tail of 40 chunks per
run), _broadcastAgentToolEvent and _broadcastAgentToolStoredChunks (replay only running runs and the newest
20 ended ones; older runs' chunks are not even read from the child). We also run our own workers' workers with eventDelivery: "terminal" because nobody watches their chunks live.
Activity