Repository navigation
Think recovery bypasses onChatError for beforeTurn failures and replaces the original error #2551
Description
Describe the bug
An application error thrown by beforeTurn is handled differently depending on whether the turn is a normal WebSocket request or a recovered turn. Recovery bypasses onChatError, emits onExhausted with reason: "recovery_error", and replaces the submission's original error with the configured terminalMessage.
This loses both the application's user-facing error mapping and the cause needed to classify the failed submission. The original error is only logged to the console.
To reproduce
Override beforeTurn to throw new Error("AccessDenied"), and onChatError to return new Error("Please sign in again."). Set chatRecovery.terminalMessage to "Recovery stopped.".
Run a normal WebSocket turn, then a recovery continuation and a recovery retry on separate instances. Reproduced with real SQLite-backed Durable Objects in Miniflare; no model or external service is needed:
Both recovery cases emit onExhausted on attempt 1, with attempts still available.
Standalone reproduction
The probe seeds an interrupted submission and calls the recovery callback body directly to make the failure deterministic without forcing an isolate eviction. The ordinary WebSocket path is the control case.
In an empty directory, create package.json with { "private": true, "type": "module" }, then install:
pnpm add --ignore-scripts @cloudflare/think@0.20.2 agents@0.28.0 ai@7.0.0 zod@4.5.4
pnpm add -D --ignore-scripts esbuild@0.28.1 miniflare@5.20261001.0-alphaprobe.js:
import { Think } from '@cloudflare/think';
import { getAgentByName } from 'agents';
export class RecoveryProbe extends Think {
workspaceBash = false;
includeMcpTools = false;
errors = [];
exhausted = [];
beforeTurnCalls = 0;
chatRecovery = {
terminalMessage: 'Recovery stopped.',
onExhausted: (ctx) => { this.exhausted.push(ctx); }
};
getModel() {
return {
specificationVersion: 'v4', provider: 'probe', modelId: 'unused', supportedUrls: {},
doStream: async () => { throw new Error('Unexpected model call'); },
doGenerate: async () => { throw new Error('Unexpected model call'); }
};
}
beforeTurn() {
this.beforeTurnCalls++;
throw new Error('AccessDenied');
}
onChatError(error) {
this.errors.push(String(error));
return new Error('Please sign in again.');
}
async verify(kind) {
const user = { id: 'user', role: 'user', parts: [{ type: 'text', text: 'hello' }] };
if (kind === 'normal') {
await this._handleChatRequest({ id: 'client', send() {} }, {
type: 'chat-request', id: 'root', init: { body: JSON.stringify({ messages: [user] }) }
});
} else {
await this.addMessages([user, ...(kind === 'continue' ? [
{ id: 'assistant', role: 'assistant', parts: [{ type: 'text', text: 'partial' }] }
] : [])]);
this._ensureSubmissionTable();
this.sql`INSERT INTO cf_think_submissions
(submission_id, idempotency_key, request_id, status, messages_json, created_at)
VALUES ('root', 'root', 'root', 'running', '[]', ${Date.now()})`;
const incidentId = 'repro';
await this.ctx.storage.put('cf:chat-recovery:incident:repro', {
incidentId, requestId: 'root', recoveryRootRequestId: 'root',
recoveryKind: kind, attempt: 1, maxAttempts: 10, status: 'attempting',
firstSeenAt: Date.now(), lastAttemptAt: Date.now()
});
const data = { originalRequestId: 'root', recoveredRequestId: 'root', incidentId };
if (kind === 'continue') await this._chatRecoveryContinueDetached(data);
else await this._chatRecoveryRetryDetached(data);
}
return {
beforeTurnCalls: this.beforeTurnCalls,
onChatError: this.errors,
onExhausted: this.exhausted,
submission: await this.inspectSubmission('root'),
terminal: await this.ctx.storage.get('cf:chat:last-terminal')
};
}
}
export default {
async fetch(request, env) {
const kind = new URL(request.url).pathname.slice(1);
const agent = await getAgentByName(env.PROBE, kind);
return Response.json(await agent.verify(kind));
}
};run.mjs:
import assert from 'node:assert/strict';
import { readFile, writeFile } from 'node:fs/promises';
import { builtinModules } from 'node:module';
import { build } from 'esbuild';
import { Miniflare, Log, LogLevel, convertV4MiniflareOptions } from 'miniflare';
await build({ entryPoints: ['probe.js'], outfile: 'bundle.mjs', bundle: true,
format: 'esm', platform: 'browser', target: 'es2022',
conditions: ['workerd', 'worker', 'browser'], external: ['cloudflare:*', 'node:*', ...builtinModules] });
const mf = new Miniflare(convertV4MiniflareOptions({ workers: [{
name: 'repro', modules: true, script: await readFile('bundle.mjs', 'utf8'),
compatibilityDate: '2026-03-05', compatibilityFlags: ['nodejs_compat', 'enhanced_error_serialization', 'enable_request_signal'],
durableObjects: { PROBE: { className: 'RecoveryProbe', useSQLite: true } }
}], log: new Log(LogLevel.ERROR) }));
try {
const results = {};
for (const kind of ['normal', 'continue', 'retry']) {
const response = await mf.dispatchFetch(`http://repro/${kind}`);
const body = await response.text();
assert.equal(response.status, 200, body);
const result = JSON.parse(body);
results[kind] = result;
console.log(kind, JSON.stringify(result));
}
await writeFile('result.json', JSON.stringify(results, null, 2));
} finally { await mf.dispose(); }Run node run.mjs. Results are printed and saved to result.json.
Expected behavior
Application errors during a recovered turn should go through the existing onChatError hook, preserving its mapped terminal message for clients and reconnects. The underlying cause should remain available on the failed submission. The turn must still terminate; platform-transient retries and actual recovery-budget exhaustion should keep their existing behavior.
Version
@cloudflare/think@0.20.2, agents@0.28.0.
Additional context
_handleRecoveryCallbackError routes ordinary errors to _exhaustRecoveryGiveUp without calling onChatError. _exhaustChatRecovery then stores terminalMessage as the submission error.
Related: #1659 added this terminalization path to prevent a thrown recovery callback from leaving the turn stuck. That guarantee is necessary; the remaining issue is preserving the application's error handling when terminalizing.
Activity