Repository navigation
fix(chat-recovery): seal a turn after repeated interrupted attempts - #2615
Conversation
…ed (#2613) A Durable Object's own memory-limit reset makes every later storage call in that invocation reject, so the OOM counters written on the failure path (recordOomAndDecide, the alarm breaker's strike) never advanced and recovery looped forever. Count each interrupted attempt in beginIncident when the next isolate replays the interrupted turn run, before anything heavy runs, and confirm the write with storage.sync(). Seal at chatRecovery.maxInterruptedAttempts (default 5) with interrupted_attempts_exceeded. Progress and debounce never reset the count. Remove the failure-path OOM bookkeeping from Think and AIChatAgent. maxOomRetries stays as a deprecated alias (maxOomRetries + 1).
🔴 agents import sizes: 2 entry points over threshold
Changed exports (7)
How this worksEach runtime export is bundled on its own, minified, and gzipped. Changes smaller than 100 B, or smaller than 1% and 1 KiB, are ignored. Growth over 10% or 5 KiB is marked 🔴. This report is informational and does not fail CI. The workflow artifact contains every measurement. Compared |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| // A recovered fiber is a run whose isolate died before it finished, so | ||
| // this opening counts one interrupted attempt for the turn (#2613). | ||
| const { incident, config, exhausted } = await this.beginIncident({ | ||
| requestId, | ||
| recoveryRootRequestId, | ||
| latestUserMessageId: snapshot?.latestUserMessageId, | ||
| recoveryKind | ||
| recoveryKind, | ||
| trigger: { kind: "interrupted", runId: ctx.id } |
There was a problem hiding this comment.
🔴 Oversized partials bypass the interruption cap
If getPartialStreamText exceeds the memory limit, beginIncident never counts the recovered run. Every replay reconstructs the same partial before the new budget runs, leaving the crash loop unbounded.
Learn more
On wake, the engine reconstructs the orphaned assistant response from its durable stream log before it opens the incident. The hosts' partial reconstruction reads every stored chunk and decodes them into parts; a sufficiently large log can itself exhaust isolate memory. Such a reset occurs before the added beginIncident call, so no count is written even though the turn is retried.
Example: An interrupted turn has enough buffered chunks that rebuilding its partial exceeds the isolate limit. Each new isolate attempts the same reconstruction, dies, and leaves interruptedAttempts unchanged.
Recommended fix: Open and sync an incident using the recovered run ID before loading or decoding the stream log. Defer reconstruction and classification until after the durable interruption count and exhaustion check, while preserving the final recovery kind needed for terminalization.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Not fixed in this PR. A reset while rebuilding the partial happens before the replay opens the incident, so it is not counted. It does not re-run the model, and bounding it needs a terminal path that skips rebuilding the partial, since sealing through the normal path would hit the same reset. It is listed under what remains unbounded in design/rfc-chat-recovery-interrupted-attempts.md.
| const legacyRetries = custom?.maxOomRetries; | ||
| if (typeof legacyRetries === "number" && legacyRetries >= 0) { | ||
| return Math.floor(legacyRetries) + 1; | ||
| } |
There was a problem hiding this comment.
📝 Info: Legacy OOM setting also bounds deploys
Existing maxOomRetries configurations now limit all interrupted turns, including deploy resets. Operators relying on deploy tolerance need to revisit that setting.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Intended. The maxOomRetries JSDoc and the docs table say it now counts every interrupted attempt, deploys included, and the changeset marks it deprecated.
agents
@cloudflare/ai-chat
@cloudflare/codemode
hono-agents
@cloudflare/shell
@cloudflare/think
@cloudflare/voice
@cloudflare/worker-bundler
commit: |
…ehind Every attempt that can run heavy work (the replay of an interrupted turn run, and each recovery callback through ChatRecoveryEngine.runRecoveryAttempt) records itself on the incident as attemptInFlight, confirmed with storage.sync(), and clears it when it finishes. The next attempt counts a record left by another isolate. This also catches a callback that dies before starting its turn, which leaves no turn run to replay. Move the decision record into its own RFC, update the living design docs, and shorten the changeset to one line.
| const { attemptInFlight: _previous, ...rest } = existing; | ||
| return { | ||
| incident: { | ||
| ...rest, | ||
| ...(interruptedAttempts > 0 ? { interruptedAttempts } : {}), | ||
| ...(exhausted ? {} : { attemptInFlight: attempt }) | ||
| }, |
There was a problem hiding this comment.
🔴 Overlapping attempts lose interrupted replay counts
When a callback finishes while its replay still runs, attemptInFlight no longer records the replay. A reset of that replay then leaves no death to count, allowing repeated work past the limit.
Learn more
A recovered turn replay can enqueue a callback before it finishes. The callback's admission replaces the replay's sole persisted marker. If the callback finishes first, #finishAttempt removes that marker, although the replay is still active. If the replay then resets, the next isolate sees no in-flight marker and does not count that interrupted replay.
Example: Replay A opens an incident, callback B starts in the same object, B completes and clears its marker, then A hits a memory reset. Replay C finds no marker from A and starts without consuming another interruption.
Recommended fix: Track overlapping attempts independently, or preserve the older live replay marker when the callback finishes. Ensure the next admission counts each dead attempt without counting a still-running attempt from the same object.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed in eb6ec86: attemptsInFlight is now a list. Entries from this isolate are attempts still running and are kept; each other isolate that left entries counts once, however many it left. Covered by tracks overlapping attempts apart, so a replay that dies after its callback finished still counts and counts one interruption for an isolate that died running two attempts.
| const { incident, exhausted } = admitChatRecoveryAttempt({ | ||
| existing, | ||
| attempt, | ||
| config: adapter.resolveConfig(), | ||
| awaitingClientInteraction: | ||
| adapter.isAwaitingClientInteraction?.() ?? false | ||
| }); |
There was a problem hiding this comment.
🔍 Recovery callbacks also consume the budget
The PR description names turn replay as the counting point, but runRecoveryAttempt also counts dead callbacks before a turn starts. The description can mislead readers about which failures exhaust recovery.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Updated the PR description: it names both counting points, the replay of an interrupted turn run and each recovery callback.
mattzcarey
changed the title
…p sealed callbacks attemptsInFlight is now a list, so a replay and the callback it schedules in the same isolate are tracked apart; each dead isolate counts once. Removing a finished attempt's entry is confirmed with storage.sync(), so a reset right after a clean finish cannot look like an interruption. A callback still queued for a sealed incident does nothing instead of re-running the model.
There was a problem hiding this comment.
🟡 Accepted decision record loses its snapshot
The accepted RFC rewrites its proposal after the decision. Design rules keep accepted RFCs as snapshots; this erases the original decision history.
Learn more
An accepted RFC is a point-in-time decision record, while a design doc describes the current implementation. Design rules explicitly distinguish these roles and require RFCs not to be updated after the decision. This revision rewrites the accepted proposal and alternatives to reflect the new list-based implementation, so the earlier accepted proposal no longer survives in the record.
Example: A contributor investigating why the original single-record approach was adopted sees only the revised list-based proposal, not the original decision or its subsequent correction.
Recommended fix: Keep the accepted RFC as originally decided. Describe the list-based implementation in the living chat design document and, if its rationale merits a decision record, add a separate amendment RFC linked from that document.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
This RFC is new in this PR and has not been merged, so there is no earlier accepted version to preserve. The single-record approach was never accepted on main; it is listed under the alternatives, with why it was replaced. The decision takes effect when this PR lands, and later changes will go in a new RFC.
## pnpm-workspace.yaml (default) ## Dependency Updates | Package | From | To | Type | | --- | --- | --- | --- | | `@better-auth/cimd` | 1.7.11 | 1.7.12 | patch | | `@better-auth/mcp` | 1.7.11 | 1.7.12 | patch | | `@better-auth/oauth-provider` | 1.7.11 | 1.7.12 | patch | | `@better-auth/passkey` | 1.7.11 | 1.7.12 | patch | | `@better-auth/sso` | 1.7.11 | 1.7.12 | patch | | `@cloudflare/ai-chat` | 0.12.1 | 0.12.2 | patch | | `@inlang/plugin-message-format` | 4.5.4 | 4.5.5 | patch | | `ai` | 7.0.146 | 7.0.148 | patch | | `better-auth` | 1.7.11 | 1.7.12 | patch | | `effectful-better-auth` | 1.0.5 | 1.0.7 | patch | ## Release Notes <details> <summary><b>@<!---->better-auth/cimd</b> (1.7.11 → 1.7.12)</summary> > [!IMPORTANT] > **Action required** > > - If you use ID token sign-in with a provider that has no `clientId`, set `clientId` to every client ID your apps sign in with. ([#11931](better-auth/better-auth#11931)) ## `better-auth` ### Bug Fixes - Fixed the JWT plugin to verify JWTs and session cookie-cache JWTs with the stored key's algorithm instead of the token header's, and to warn when signing or verifying a JWT without an issuer or audience ([#11925](better-auth/better-auth#11925)) - Fixed OAuth callbacks completed through the OAuth proxy to run the same checks as the standard OAuth callback, including issuer and ID token nonce validation ([#11930](better-auth/better-auth#11930)) - Fixed `auth.api.getSession()` to keep the original database error as `error.cause` when the session lookup fails ([#11924](better-auth/better-auth#11924)) For detailed changes, see [`CHANGELOG`](https://github.com/better-auth/better-auth/blob/29735514c737d2abeac2fb84f0691fd316231949/packages/better-auth/CHANGELOG.md) ## `@better-auth/core` ### Bug Fixes - Fixed `mapProfileToUser` failing to typecheck under `exactOptionalPropertyTypes` when it returns `undefined` for `name`, `email`, or `image` ([#11918](better-auth/better-auth#11918)) - Fixed ID token sign-in to reject tokens when the provider has no client ID, since their audience cannot be checked ([#11931](better-auth/better-auth#11931)) - Fixed a custom Facebook `getUserInfo` to no longer receive client-submitted access tokens issued to another Facebook app, and ID tokens missing `exp` or `iat` are now rejected ([#11919](better-auth/better-auth#11919)) - Restored the Twitch and Paybin ID token handling from before 1.7.11, which made account info fail once the ID token stored at sign-in expired ([#11920](https://github.com/better-auth/bette …[full notes](https://github.com/better-auth/better-auth/releases/tag/v1.7.12) </details> <details> <summary><b>@<!---->better-auth/mcp</b> (1.7.11 → 1.7.12)</summary> > [!IMPORTANT] > **Action required** > > - If you use ID token sign-in with a provider that has no `clientId`, set `clientId` to every client ID your apps sign in with. ([#11931](better-auth/better-auth#11931)) ## `better-auth` ### Bug Fixes - Fixed the JWT plugin to verify JWTs and session cookie-cache JWTs with the stored key's algorithm instead of the token header's, and to warn when signing or verifying a JWT without an issuer or audience ([#11925](better-auth/better-auth#11925)) - Fixed OAuth callbacks completed through the OAuth proxy to run the same checks as the standard OAuth callback, including issuer and ID token nonce validation ([#11930](better-auth/better-auth#11930)) - Fixed `auth.api.getSession()` to keep the original database error as `error.cause` when the session lookup fails ([#11924](better-auth/better-auth#11924)) For detailed changes, see [`CHANGELOG`](https://github.com/better-auth/better-auth/blob/29735514c737d2abeac2fb84f0691fd316231949/packages/better-auth/CHANGELOG.md) ## `@better-auth/core` ### Bug Fixes - Fixed `mapProfileToUser` failing to typecheck under `exactOptionalPropertyTypes` when it returns `undefined` for `name`, `email`, or `image` ([#11918](better-auth/better-auth#11918)) - Fixed ID token sign-in to reject tokens when the provider has no client ID, since their audience cannot be checked ([#11931](better-auth/better-auth#11931)) - Fixed a custom Facebook `getUserInfo` to no longer receive client-submitted access tokens issued to another Facebook app, and ID tokens missing `exp` or `iat` are now rejected ([#11919](better-auth/better-auth#11919)) - Restored the Twitch and Paybin ID token handling from before 1.7.11, which made account info fail once the ID token stored at sign-in expired ([#11920](https://github.com/better-auth/bette …[full notes](https://github.com/better-auth/better-auth/releases/tag/v1.7.12) </details> <details> <summary><b>@<!---->better-auth/oauth-provider</b> (1.7.11 → 1.7.12)</summary> > [!IMPORTANT] > **Action required** > > - If you use ID token sign-in with a provider that has no `clientId`, set `clientId` to every client ID your apps sign in with. ([#11931](better-auth/better-auth#11931)) ## `better-auth` ### Bug Fixes - Fixed the JWT plugin to verify JWTs and session cookie-cache JWTs with the stored key's algorithm instead of the token header's, and to warn when signing or verifying a JWT without an issuer or audience ([#11925](better-auth/better-auth#11925)) - Fixed OAuth callbacks completed through the OAuth proxy to run the same checks as the standard OAuth callback, including issuer and ID token nonce validation ([#11930](better-auth/better-auth#11930)) - Fixed `auth.api.getSession()` to keep the original database error as `error.cause` when the session lookup fails ([#11924](better-auth/better-auth#11924)) For detailed changes, see [`CHANGELOG`](https://github.com/better-auth/better-auth/blob/29735514c737d2abeac2fb84f0691fd316231949/packages/better-auth/CHANGELOG.md) ## `@better-auth/core` ### Bug Fixes - Fixed `mapProfileToUser` failing to typecheck under `exactOptionalPropertyTypes` when it returns `undefined` for `name`, `email`, or `image` ([#11918](better-auth/better-auth#11918)) - Fixed ID token sign-in to reject tokens when the provider has no client ID, since their audience cannot be checked ([#11931](better-auth/better-auth#11931)) - Fixed a custom Facebook `getUserInfo` to no longer receive client-submitted access tokens issued to another Facebook app, and ID tokens missing `exp` or `iat` are now rejected ([#11919](better-auth/better-auth#11919)) - Restored the Twitch and Paybin ID token handling from before 1.7.11, which made account info fail once the ID token stored at sign-in expired ([#11920](https://github.com/better-auth/bette …[full notes](https://github.com/better-auth/better-auth/releases/tag/v1.7.12) </details> <details> <summary><b>@<!---->better-auth/passkey</b> (1.7.11 → 1.7.12)</summary> > [!IMPORTANT] > **Action required** > > - If you use ID token sign-in with a provider that has no `clientId`, set `clientId` to every client ID your apps sign in with. ([#11931](better-auth/better-auth#11931)) ## `better-auth` ### Bug Fixes - Fixed the JWT plugin to verify JWTs and session cookie-cache JWTs with the stored key's algorithm instead of the token header's, and to warn when signing or verifying a JWT without an issuer or audience ([#11925](better-auth/better-auth#11925)) - Fixed OAuth callbacks completed through the OAuth proxy to run the same checks as the standard OAuth callback, including issuer and ID token nonce validation ([#11930](better-auth/better-auth#11930)) - Fixed `auth.api.getSession()` to keep the original database error as `error.cause` when the session lookup fails ([#11924](better-auth/better-auth#11924)) For detailed changes, see [`CHANGELOG`](https://github.com/better-auth/better-auth/blob/29735514c737d2abeac2fb84f0691fd316231949/packages/better-auth/CHANGELOG.md) ## `@better-auth/core` ### Bug Fixes - Fixed `mapProfileToUser` failing to typecheck under `exactOptionalPropertyTypes` when it returns `undefined` for `name`, `email`, or `image` ([#11918](better-auth/better-auth#11918)) - Fixed ID token sign-in to reject tokens when the provider has no client ID, since their audience cannot be checked ([#11931](better-auth/better-auth#11931)) - Fixed a custom Facebook `getUserInfo` to no longer receive client-submitted access tokens issued to another Facebook app, and ID tokens missing `exp` or `iat` are now rejected ([#11919](better-auth/better-auth#11919)) - Restored the Twitch and Paybin ID token handling from before 1.7.11, which made account info fail once the ID token stored at sign-in expired ([#11920](https://github.com/better-auth/bette …[full notes](https://github.com/better-auth/better-auth/releases/tag/v1.7.12) </details> <details> <summary><b>@<!---->better-auth/sso</b> (1.7.11 → 1.7.12)</summary> > [!IMPORTANT] > **Action required** > > - If you use ID token sign-in with a provider that has no `clientId`, set `clientId` to every client ID your apps sign in with. ([#11931](better-auth/better-auth#11931)) ## `better-auth` ### Bug Fixes - Fixed the JWT plugin to verify JWTs and session cookie-cache JWTs with the stored key's algorithm instead of the token header's, and to warn when signing or verifying a JWT without an issuer or audience ([#11925](better-auth/better-auth#11925)) - Fixed OAuth callbacks completed through the OAuth proxy to run the same checks as the standard OAuth callback, including issuer and ID token nonce validation ([#11930](better-auth/better-auth#11930)) - Fixed `auth.api.getSession()` to keep the original database error as `error.cause` when the session lookup fails ([#11924](better-auth/better-auth#11924)) For detailed changes, see [`CHANGELOG`](https://github.com/better-auth/better-auth/blob/29735514c737d2abeac2fb84f0691fd316231949/packages/better-auth/CHANGELOG.md) ## `@better-auth/core` ### Bug Fixes - Fixed `mapProfileToUser` failing to typecheck under `exactOptionalPropertyTypes` when it returns `undefined` for `name`, `email`, or `image` ([#11918](better-auth/better-auth#11918)) - Fixed ID token sign-in to reject tokens when the provider has no client ID, since their audience cannot be checked ([#11931](better-auth/better-auth#11931)) - Fixed a custom Facebook `getUserInfo` to no longer receive client-submitted access tokens issued to another Facebook app, and ID tokens missing `exp` or `iat` are now rejected ([#11919](better-auth/better-auth#11919)) - Restored the Twitch and Paybin ID token handling from before 1.7.11, which made account info fail once the ID token stored at sign-in expired ([#11920](https://github.com/better-auth/bette …[full notes](https://github.com/better-auth/better-auth/releases/tag/v1.7.12) </details> <details> <summary><b>@<!---->cloudflare/ai-chat</b> (0.12.1 → 0.12.2)</summary> ### Patch Changes - [#2615](cloudflare/agents#2615) [`05843e4`](cloudflare/agents@05843e4) Thanks [@<!---->mattzcarey](https://github.com/mattzcarey)! - Chat recovery now seals a turn after `chatRecovery.maxInterruptedAttempts` interrupted attempts (default 5), so a turn that exceeds the Durable Object memory limit on every run no longer retries forever; `maxOomRetries` is deprecated. See [Stream Recovery](https://github.com/cloudflare/agents/blob/main/docs/agents/chat-agents.md#stream-recovery). </details> <details> <summary><b>better-auth</b> (1.7.11 → 1.7.12)</summary> > [!IMPORTANT] > **Action required** > > - If you use ID token sign-in with a provider that has no `clientId`, set `clientId` to every client ID your apps sign in with. ([#11931](better-auth/better-auth#11931)) ## `better-auth` ### Bug Fixes - Fixed the JWT plugin to verify JWTs and session cookie-cache JWTs with the stored key's algorithm instead of the token header's, and to warn when signing or verifying a JWT without an issuer or audience ([#11925](better-auth/better-auth#11925)) - Fixed OAuth callbacks completed through the OAuth proxy to run the same checks as the standard OAuth callback, including issuer and ID token nonce validation ([#11930](better-auth/better-auth#11930)) - Fixed `auth.api.getSession()` to keep the original database error as `error.cause` when the session lookup fails ([#11924](better-auth/better-auth#11924)) For detailed changes, see [`CHANGELOG`](https://github.com/better-auth/better-auth/blob/29735514c737d2abeac2fb84f0691fd316231949/packages/better-auth/CHANGELOG.md) ## `@better-auth/core` ### Bug Fixes - Fixed `mapProfileToUser` failing to typecheck under `exactOptionalPropertyTypes` when it returns `undefined` for `name`, `email`, or `image` ([#11918](better-auth/better-auth#11918)) - Fixed ID token sign-in to reject tokens when the provider has no client ID, since their audience cannot be checked ([#11931](better-auth/better-auth#11931)) - Fixed a custom Facebook `getUserInfo` to no longer receive client-submitted access tokens issued to another Facebook app, and ID tokens missing `exp` or `iat` are now rejected ([#11919](better-auth/better-auth#11919)) - Restored the Twitch and Paybin ID token handling from before 1.7.11, which made account info fail once the ID token stored at sign-in expired ([#11920](https://github.com/better-auth/bette …[full notes](https://github.com/better-auth/better-auth/releases/tag/v1.7.12) </details> --- *This PR was auto-generated by [catalog-update-action](https://github.com/brandhaug/catalog-update-action).* Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## pnpm-workspace.yaml (default) ## Dependency Updates | Package | From | To | Type | | --- | --- | --- | --- | | `agents` | 0.28.1 | 0.29.0 | minor | ## Release Notes <details> <summary><b>agents</b> (0.28.1 → 0.29.0)</summary> ### Minor Changes - [#2560](cloudflare/agents#2560) [`8c720d3`](cloudflare/agents@8c720d3) Thanks [@<!---->mattzcarey](https://github.com/mattzcarey)! - Agent Skills read and write a `@cloudflare/computer` Workspace, for both projection and `skills.runner()`. See [Upgrading to a Computer workspace](https://github.com/cloudflare/agents/blob/main/docs/think/index.md#upgrading-to-a-computer-workspace). ### Patch Changes - [#2584](cloudflare/agents#2584) [`3b3a88e`](cloudflare/agents@3b3a88e) Thanks [@<!---->ben-reitz](https://github.com/ben-reitz)! - `Browser` creates and closes its Browser Run sessions with the binding's typed `acquire()` and `closeSession()`, falling back to the HTTP API in fully local dev. See [Browser](https://developers.cloudflare.com/agents/tools/browser/). - [#2615](cloudflare/agents#2615) [`05843e4`](cloudflare/agents@05843e4) Thanks [@<!---->mattzcarey](https://github.com/mattzcarey)! - Chat recovery now seals a turn after `chatRecovery.maxInterruptedAttempts` interrupted attempts (default 5), so a turn that exceeds the Durable Object memory limit on every run no longer retries forever; `maxOomRetries` is deprecated. See [Stream Recovery](https://github.com/cloudflare/agents/blob/main/docs/agents/chat-agents.md#stream-recovery). - [#2604](cloudflare/agents#2604) [`5d6f5d6`](cloudflare/agents@5d6f5d6) Thanks [@<!---->mattzcarey](https://github.com/mattzcarey)! - `createAI` takes `{ accountId, apiToken }` in place of `binding`, to use a gateway in another account or to run outside a Worker. See [the models docs](https://github.com/cloudflare/agents/blob/main/docs/agents/models.md#account-id-and-api-token). - [#2622](https://github. …[full notes](https://github.com/cloudflare/agents/releases/tag/agents%400.29.0) </details> --- *This PR was auto-generated by [catalog-update-action](https://github.com/brandhaug/catalog-update-action).* Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Chat recovery now seals a turn with
interrupted_attempts_exceededoncechatRecovery.maxInterruptedAttemptsof its attempts have died (default 5, the original turn included). Each attempt (the replay of an interrupted turn run, and each_chatRecoveryContinue/_chatRecoveryRetrycallback) records itself on the incident asattemptInFlightbefore doing anything heavy, confirms it withstorage.sync(), and clears it when it finishes. The next attempt counts a record left by another isolate, and nothing is written on the failure path any more. Fixes #2613.A Durable Object's own memory-limit reset makes every later storage call in that invocation reject, and every #1825 counter was written after the reset:
recordOomAndDecidefailed on its first read, sooomAttemptsnever movedget/putthrew and fell back tostrikes = 1, so one DO logged "strike 1/3" 10,800 times a day for eight daysTests ran:
pnpm run check, the agentschatproject, and the Think and ai-chatworkerssuites. I also deployed a Think agent whose model exceeds 128 MB on every run to a personal account. On releasedagents@0.28.1it re-ran the model 14 times in 6 minutes and kept going (incident stillscheduled,attempt: 1,oomAttemptsnever written). On this branch it ran the model 5 times and sealed withinterrupted_attempts_exceeded.