What happened
Deleting a project while clients are still using it returns 204 and then never completes. The background deletion worker refuses the episodic half because the cached memory instance is in use, logs the failure, and drops the job; nothing retries it. The project stays in delete status: it keeps serving reads and writes with 200 from the process-local instance cache, its segment-store and vector-store data stays on disk, and a later POST /projects/delete is a no-op because the status is already Deleted. Only a server restart, which re-queues deleted sessions at boot, completes the deletion.
The deletion is also partial by then. The worker gathers the episode-store delete and the episodic-memory delete concurrently, so the episode-store rows are gone before the episodic branch raises. Searches on the stuck project then return fewer results and log "search_scored dropped N episode(s) found in the event index but missing from EpisodeStorage (likely index/storage drift)".
Reproduced on the event backend with SQLite and again with PostgreSQL: four concurrent clients alternating POST /memories and POST /memories/search on one project, and one POST /projects/delete issued 150 ms in. Every one of the 120 client requests returned 200, the delete returned 204, and the server logged:
[ERROR] memmachine_server.main.memmachine - Failed to delete session manual_org/race_proj
...
memmachine_server.common.errors.SessionInUseError: Session 'manual_org/race_proj' is in use and can't be deleted. Reference count: 1.
Afterwards the sessions row read ('manual_org/race_proj', 'delete'), the project's 65 segment rows and vectors were still present, a search on it returned 200 with results, and the drift warnings above appeared on every search. After a restart the boot-time re-queue deleted it ("Deleted session manual_org/race_proj").
Line numbers as of 231ce171 (main):
MemMachine.delete_session (packages/server/src/memmachine_server/main/memmachine.py:582) marks the session Deleted and enqueues it; it returns early on a session already in that status (line 601), so the API cannot re-trigger a failed deletion._delete_session_worker (line 342) catches any exception from _delete_queued_session, logs it, and moves on: no retry, no backoff._delete_queued_session (line 357) gathers _delete_session_episode_store (line 369) and _delete_session_episodic_memory (line 386) concurrently, so the episode rows are deleted whether or not the episodic branch succeeds.EpisodicMemoryManager.delete_episodic_session (episodic_memory/episodic_memory_manager.py:286) raises SessionInUseError when the instance's reference count is positive (lines 299 to 302), which is the normal state of a project with in-flight requests.start (main/memmachine.py:411) re-queues sessions in Deleted status at boot, which is why a restart heals it.
Expected
A deletion that has been acknowledged with 204 must complete on its own. Either the worker retries a session that is in use (with a backoff, since in-flight requests finish quickly), or the in-use check is replaced by draining: refuse new opens for a session in Deleted status and delete once the reference count reaches zero. In both cases the destructive steps should not run until the whole deletion can, so a refused deletion leaves the project either intact or gone, not with its episode rows missing and its index and segments present.
Notes
Found by a manual end-to-end run against the server. The refusal is the correct guard for an operator-initiated delete of a session someone is actively using; the defect is that the queued path treats the guard as a terminal failure and that the API reports success before the guard has run. Related: #1575 (a different terminal failure in the same worker) and #1576 (a stray search after the deletion finally completes recreates the project).
Investigated and written by Claude (Claude Code), filed from the account of the user who commissioned the investigation.
What happened
Deleting a project while clients are still using it returns 204 and then never completes. The background deletion worker refuses the episodic half because the cached memory instance is in use, logs the failure, and drops the job; nothing retries it. The project stays in
deletestatus: it keeps serving reads and writes with 200 from the process-local instance cache, its segment-store and vector-store data stays on disk, and a laterPOST /projects/deleteis a no-op because the status is alreadyDeleted. Only a server restart, which re-queues deleted sessions at boot, completes the deletion.The deletion is also partial by then. The worker gathers the episode-store delete and the episodic-memory delete concurrently, so the episode-store rows are gone before the episodic branch raises. Searches on the stuck project then return fewer results and log "search_scored dropped N episode(s) found in the event index but missing from EpisodeStorage (likely index/storage drift)".
Reproduced on the event backend with SQLite and again with PostgreSQL: four concurrent clients alternating
POST /memoriesandPOST /memories/searchon one project, and onePOST /projects/deleteissued 150 ms in. Every one of the 120 client requests returned 200, the delete returned 204, and the server logged:Afterwards the
sessionsrow read('manual_org/race_proj', 'delete'), the project's 65 segment rows and vectors were still present, a search on it returned 200 with results, and the drift warnings above appeared on every search. After a restart the boot-time re-queue deleted it ("Deleted session manual_org/race_proj").Line numbers as of
231ce171(main):MemMachine.delete_session(packages/server/src/memmachine_server/main/memmachine.py:582) marks the sessionDeletedand enqueues it; it returns early on a session already in that status (line 601), so the API cannot re-trigger a failed deletion._delete_session_worker(line 342) catches any exception from_delete_queued_session, logs it, and moves on: no retry, no backoff._delete_queued_session(line 357) gathers_delete_session_episode_store(line 369) and_delete_session_episodic_memory(line 386) concurrently, so the episode rows are deleted whether or not the episodic branch succeeds.EpisodicMemoryManager.delete_episodic_session(episodic_memory/episodic_memory_manager.py:286) raisesSessionInUseErrorwhen the instance's reference count is positive (lines 299 to 302), which is the normal state of a project with in-flight requests.start(main/memmachine.py:411) re-queues sessions inDeletedstatus at boot, which is why a restart heals it.Expected
A deletion that has been acknowledged with 204 must complete on its own. Either the worker retries a session that is in use (with a backoff, since in-flight requests finish quickly), or the in-use check is replaced by draining: refuse new opens for a session in
Deletedstatus and delete once the reference count reaches zero. In both cases the destructive steps should not run until the whole deletion can, so a refused deletion leaves the project either intact or gone, not with its episode rows missing and its index and segments present.Notes
Found by a manual end-to-end run against the server. The refusal is the correct guard for an operator-initiated delete of a session someone is actively using; the defect is that the queued path treats the guard as a terminal failure and that the API reports success before the guard has run. Related: #1575 (a different terminal failure in the same worker) and #1576 (a stray search after the deletion finally completes recreates the project).
Investigated and written by Claude (Claude Code), filed from the account of the user who commissioned the investigation.