PiHarness retries a session forever when its turn exceeds the Durable Object memory limit · Issue #2625 · cloudflare/agents · GitHub
Skip to content

PiHarness retries a session forever when its turn exceeds the Durable Object memory limit #2625

Open

Description

@mattzcarey

A pi session whose generation exceeds the Durable Object's 128 MB memory limit on every run is retried forever, re-running the model call each time. This is the PiHarness version of #2613, which #2615 fixes for Think and AIChatAgent chat recovery. I found it by reading the code and haven't reproduced it.

Why nothing stops it

  • pi-durable restarts the task without counting it. It commits a task as running before each phase. On the next open, its scheduler resets every running task to pending (dist/harness/scheduler.js, open()), and PiHarness.#doOpen() calls pi.resume() to run it again (harness.ts#L531-L537). The generation task's attempt only goes up when the model returns an error (the retry phase), not when the isolate dies.
  • The breaker can't count it. The session's wake job is pushed with recoveryLoop: true (harness.ts#L446-L455), so its only bound is the Lifecycle alarm memory-limit breaker. That breaker writes its strike after the reset, and a Durable Object's own reset makes every later storage call in that invocation reject, so the count never gets past 1 (Alarm memory-limit breaker (#1825) never trips after a DO memory-limit reset: strike counter can't be persisted, recovery loops indefinitely #2613). The generation also runs as tracked alarm work after the wake returns, so the breaker may not see the reset at all.
  • A seal wouldn't hold anyway. PiHarness has no onMemoryLimit hook. If the breaker did seal and purge the wake job, onStart pushes a new wake for every session with live pi tasks (harness.ts#L250-L258), and the next start resumes the same task.

What #2615 does for chat

An isolate killed by a memory-limit reset can't record anything, so chat recovery counts the death from the next isolate. Each attempt writes { isolate, token } to the incident and waits for storage.sync() before doing anything heavy, then removes it when it finishes. The next attempt counts entries left by another isolate, and the turn is sealed after maxInterruptedAttempts (default 5). See design/rfc-chat-recovery-interrupted-attempts.md in #2615.

Possible fixes

  • In pi-durable, upstream: count restarts per task where open() resets running to pending. That commit already runs in a fresh isolate before anything resumes. Past a limit, terminate the task (for example aborted with a new reason) instead of resuming it.
  • In PiHarness: record a per-session attempt entry with sync() before #wakeStep waits on live tasks, and remove it when the wait ends. On the next wake or onStart, count entries from another isolate. Past a limit, abort the session's live tasks instead of resuming them.

Either way, the next isolate can't tell a memory-limit reset from a deploy, so a deploy also counts, as it does in #2615.

Testing

A unit test can fake the reset by leaving the attempt entry, or the running task, behind and opening a new harness. The real check needs a deploy, because local workerd doesn't enforce 128 MB. The deployed test for #2615 used a model that allocates past the limit on every run and counted model runs in a separate Durable Object: released agents kept going at 14 runs in 6 minutes, and the fix stopped at 5.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions