You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
PiHarness retries a session forever when its turn exceeds the Durable Object memory limit #2625
A pi session whose generation exceeds the Durable Object's 128 MB memory limit on every run is retried forever, re-running the model call each time. This is the PiHarness version of #2613, which #2615 fixes for Think and AIChatAgent chat recovery. I found it by reading the code and haven't reproduced it.
Why nothing stops it
pi-durable restarts the task without counting it. It commits a task as running before each phase. On the next open, its scheduler resets every running task to pending (dist/harness/scheduler.js, open()), and PiHarness.#doOpen() calls pi.resume() to run it again (harness.ts#L531-L537). The generation task's attempt only goes up when the model returns an error (the retry phase), not when the isolate dies.
A seal wouldn't hold anyway. PiHarness has no onMemoryLimit hook. If the breaker did seal and purge the wake job, onStart pushes a new wake for every session with live pi tasks (harness.ts#L250-L258), and the next start resumes the same task.
An isolate killed by a memory-limit reset can't record anything, so chat recovery counts the death from the next isolate. Each attempt writes { isolate, token } to the incident and waits for storage.sync() before doing anything heavy, then removes it when it finishes. The next attempt counts entries left by another isolate, and the turn is sealed after maxInterruptedAttempts (default 5). See design/rfc-chat-recovery-interrupted-attempts.md in #2615.
Possible fixes
In pi-durable, upstream: count restarts per task where open() resets running to pending. That commit already runs in a fresh isolate before anything resumes. Past a limit, terminate the task (for example aborted with a new reason) instead of resuming it.
In PiHarness: record a per-session attempt entry with sync() before #wakeStep waits on live tasks, and remove it when the wait ends. On the next wake or onStart, count entries from another isolate. Past a limit, abort the session's live tasks instead of resuming them.
Either way, the next isolate can't tell a memory-limit reset from a deploy, so a deploy also counts, as it does in #2615.
Testing
A unit test can fake the reset by leaving the attempt entry, or the running task, behind and opening a new harness. The real check needs a deploy, because local workerd doesn't enforce 128 MB. The deployed test for #2615 used a model that allocates past the limit on every run and counted model runs in a separate Durable Object: released agents kept going at 14 runs in 6 minutes, and the fix stopped at 5.
Activity