Before you start

• I searched existing issues and this is not a duplicate.
• I can reproduce this on the latest main.

Problem summary

After delete_by_run() deletes one run's events, DbRunEventStore computes new seq values from the max over the remaining rows, so later puts reissue seq numbers that were already handed out and the new events become permanently invisible to after_seq cursor pagination — while MemoryRunEventStore and JsonlRunEventStore keep the seq strictly increasing for the identical operation sequence.

Affected area(s)

• Backend API (gateway / endpoints / SSE)

What happened?

DbRunEventStore assigns each new event seq = (max seq over remaining rows) + 1 — put at backend/packages/harness/deerflow/runtime/events/store/db.py:204, put_batch at db.py:232, put_if_absent at db.py:303. delete_by_run() (db.py:542-566) deletes rows without recording any per-thread high-water mark, so the max drops and the next write can:

1. reissue a seq value that was already assigned to a (now deleted) event — the assigned sequence 1, 2, 2 is not strictly increasing; and
2. store the new event at a seq that client feed cursors have already advanced past, making it invisible: list_messages(after_seq=2) returns [] for an event stored at seq 2.

The memory and JSONL backends keep a per-thread counter that delete_by_run never lowers (memory.py:34-38, counter untouched by delete_by_run at memory.py:230-243; jsonl.py:128-130, counter only popped by delete_by_thread at jsonl.py:466), so the same operation sequence yields seq 3 and the message is returned. The three implementations of one interface disagree on a documented, tested public method.

Reachability caveat, stated up front: I could not find an in-tree production caller of the run-event store's delete_by_run() today — the gateway wires delete_by_thread() (app/gateway/routers/threads.py:813), and the other delete_by_run references are different classes (FeedbackRepository.delete_by_run in backend/packages/harness/deerflow/persistence/feedback/sql.py:175). So this is a store-contract defect reachable by direct RunEventStore consumers (SDK/embedded use, future run-delete routes), not currently a live HTTP data-loss path.

Expected behavior

Per the RunEventStore ABC contract, backend/packages/harness/deerflow/runtime/events/store/base.py:49-57:

All implementations must guarantee:

1. put() events are retrievable in subsequent queries
2. seq is strictly increasing within the same thread

db.py's own put docstring (db.py:188-196) likewise says it "assign[s] a monotonic seq".

So after put(r1) → seq 1, put(r2) → seq 2, delete_by_run(t1, r2), the next put(r3) on the same thread must receive a seq greater than every seq previously handed out on that thread (3, as memory and jsonl do), and list_messages(after_seq=2) must return r3's message. The exclusive before_seq/after_seq cursors (base.py:112-119, exposed over HTTP at app/gateway/routers/runs.py:124) only stay sound if newly assigned seqs exceed every seq previously handed to a client while the thread is live.

Steps to reproduce

Save as repro.py, then cd backend && uv run python repro.py. It runs the same operation sequence against all three RunEventStore backends (sqlite for the db one).

import asyncio
import tempfile
from pathlib import Path

from deerflow.runtime.events.store.db import DbRunEventStore
from deerflow.runtime.events.store.jsonl import JsonlRunEventStore
from deerflow.runtime.events.store.memory import MemoryRunEventStore


async def scenario(store_factory, label):
    store = await store_factory()
    a1 = await store.put(thread_id="t1", run_id="r1", event_type="llm.ai.response", category="message", content={"type": "ai", "id": "a1", "content": "x"})
    a2 = await store.put(thread_id="t1", run_id="r2", event_type="llm.ai.response", category="message", content={"type": "ai", "id": "a2", "content": "x"})
    print(f"[{label}] r1 seq={a1['seq']}  r2 seq={a2['seq']}")
    await store.delete_by_run("t1", "r2", user_id=None)
    a3 = await store.put(thread_id="t1", run_id="r3", event_type="llm.ai.response", category="message", content={"type": "ai", "id": "a3", "content": "x"})
    print(f"[{label}] after delete_by_run(r2), r3 first event seq={a3['seq']}  (previously assigned max seq was 2)")
    fwd = await store.list_messages("t1", limit=50, after_seq=a2["seq"], user_id=None)
    print(f"[{label}] list_messages(after_seq={a2['seq']}) -> {[e['seq'] for e in fwd]}  (r3's message must appear)")


async def memory_factory():
    return MemoryRunEventStore()


async def jsonl_factory():
    return JsonlRunEventStore(Path(tempfile.mkdtemp()) / "jsonl")


async def db_factory():
    from deerflow.persistence.engine import get_session_factory, init_engine

    tmp = tempfile.mkdtemp()
    await init_engine("sqlite", url=f"sqlite+aiosqlite:///{Path(tmp) / 'events.db'}", sqlite_dir=str(tmp))
    return DbRunEventStore(get_session_factory())


async def main():
    await scenario(memory_factory, "memory")
    print()
    await scenario(jsonl_factory, "jsonl ")
    print()
    await scenario(db_factory, "db    ")


asyncio.run(main())

Relevant logs

Verbatim output of uv run python repro.py (identical at commit 5872d07 and at current main a4427625):

[memory] r1 seq=1  r2 seq=2
[memory] after delete_by_run(r2), r3 first event seq=3  (previously assigned max seq was 2)
[memory] list_messages(after_seq=2) -> [3]  (r3's message must appear)

[jsonl ] r1 seq=1  r2 seq=2
[jsonl ] after delete_by_run(r2), r3 first event seq=3  (previously assigned max seq was 2)
[jsonl ] list_messages(after_seq=2) -> [3]  (r3's message must appear)

[db    ] r1 seq=1  r2 seq=2
[db    ] after delete_by_run(r2), r3 first event seq=2  (previously assigned max seq was 2)
[db    ] list_messages(after_seq=2) -> []  (r3's message must appear)

The db block shows both symptoms: r3's first event reissues seq 2 (the assigned sequence 1, 2, 2 is not strictly increasing), and list_messages(after_seq=2) returns [], so r3's message is invisible to any client whose cursor already advanced past seq 2.

How are you running DeerFlow?

Other

Operating system

macOS

Platform details

arm64, zsh

Python version

Python 3.12.14

Node.js version

N/A (Python-only repro against the backend harness package)

pnpm version

N/A

uv version

0.12.8

Git state

branch: main
commit: 5872d07 — also reproduced on current main a442762; backend/packages/harness/deerflow/runtime/events/store/ is unchanged between the two commits.

Support bundle summary

Not applicable — deterministic store-logic bug, no setup/config/sandbox involvement.

Additional context

Root cause: DbRunEventStore derives each new seq from the stored rows (seq = (max_seq or 0) + 1 at db.py:204, db.py:232, db.py:303), and delete_by_run (db.py:542-566) removes rows without recording the thread's high-water mark, so the max drops. The sibling backends never lower their counters on a single-run delete: MemoryRunEventStore._seq_counters (memory.py:34-38, untouched by delete_by_run at memory.py:230-243) and JsonlRunEventStore._next_seq (jsonl.py:128-130, counter popped only by delete_by_thread at jsonl.py:466, which removes all rows and documents restarting at seq 1).

The shared contract suite passes because test_delete_by_run in backend/tests/test_run_event_store.py only puts, deletes, and reads — it never puts again after the delete, so this divergence is untested.

A possible fix direction: keep a per-thread high-water mark that deletions never lower — e.g. delete_by_run/delete_by_thread record the thread's current max seq into a durable watermark before deleting rows, and writers assign max(max_seq_in_table, watermark) + 1 — plus a contract test parametrized over backends (put r1/r2 → delete_by_run(r2) → put r3, assert r3.seq > r2.seq and that list_messages(after_seq=r2.seq) returns r3's message), which fails on db today and passes on memory/jsonl.

Related: #5530 fixed the deletion mutation fence in this same file (a concurrency race between delete and admitted writers). This report is a different, deterministic defect — the seq watermark itself drops after a delete even with no concurrency.

Happy to be assigned and open a PR with the watermark approach and the contract test.