Current Behavior

EdgeShard writes every operation to its WAL, but nothing ever acknowledges or truncates it:

• EdgeShard::update appends each operation with wal.write(...) (update.rs#L32).
• EdgeShard::flush flushes the WAL and then all segments, but never calls ack, although flush_all returns the persisted version (mod.rs#L268-L277).
• EdgeShard::load never reads the WAL back, so nothing else truncates it either.

As a result, wal/ only ever grows, and every closed segment stays forever. On Windows, each segment is allocated in full up front (segment.rs#L166), so every segment costs 32 MiB of disk space by default. Loading a shard also gets slower over time, because Wal::open reads and checks every segment.

We embed qdrant-edge in a desktop app for local RAG, with one EdgeShard per data source. Numbers from real installations:

• Windows 11: one shard held the WAL entries [0, 15451) in four segments plus a preallocated empty one, so 160 MiB of WAL. Opening these segments took 175 ms.
• macOS: in every shard, the WAL was larger than the segments, e.g., 76 MB of WAL next to 50 MB of segment data.

#10189 noted this for edge-tool ("never acked … grows with every upsert") and skips wal/ on upload. The library itself is still affected, in the published 0.8.0 and on master as of 016542a.

Steps to Reproduce

1. Create a shard with small WAL segments, so the effect shows quickly. The default 32 MiB segments behave the same.

   let config = EdgeConfig {
       wal_options: Some(WalOptions {
           segment_capacity: 8 * 1024,
           ..WalOptions::default()
       }),
       ..config
   };
   let shard = EdgeShard::new(&path, config)?;

2. Upsert a few hundred points one at a time, calling shard.flush() after each update.

3. Call shard.optimize(), drop the shard, and load it again with EdgeShard::load(&path, None).

4. List path/wal.

The directory holds one closed-* file for every filled segment. The count only goes up with further updates, flushes, optimizations, and reloads.

Expected Behavior

Once flush() has persisted the segments, the WAL acknowledges the entries up to the persisted version and truncates its closed segments, keeping retain_closed of them, like the WAL of a collection shard.

Possible Solution

Acknowledge the persisted version in EdgeShard::flush, as the collection's flush worker does after flushing its segments (flush_workers.rs#L99-L124):

pub fn flush(&self) -> OperationResult<()> {
    self.wal
        .lock()
        .flush()
        .map_err(|e| OperationError::service_error(format!("WAL flush failed: {e}")))?;

    let persisted_version = self.segments.read().flush_all(FlushMode::Sync, true)?;

    self.wal
        .lock()
        .ack(persisted_version)
        .map_err(|e| OperationError::service_error(format!("WAL ack failed: {e}")))?;

    Ok(())
}

ack truncates through prefix_truncate, which keeps the newest closed segment. That matters, see below.

Related: the WAL also carries the operation numbering

While working around this, we found that the WAL must not be deleted completely. Wal::open takes the next operation number from the first closed-* segment and starts at 0 without one (lib.rs#L179). EdgeShard::load does not restore it from the segment versions. The segments then skip every operation whose number is below the stored point version (segment_ops.rs#L641), so later upserts and deletes of existing points are silently ignored.

We confirmed this with a test: after removing all closed-* segments of a shard and loading it again, an upsert of an existing point with a new payload succeeds, but the point keeps its old payload. Keeping the newest closed segment avoids this.

This might also affect shards uploaded without wal/ since #10189 if they get updated after the download. We have not tested that flow. Perhaps load should start an empty WAL after the highest segment version, e.g., with Wal::generate_empty_wal_starting_at_index (lib.rs#L105).

Context (Environment)

• qdrant-edge 0.8.0 from crates.io, Rust 1.99.0. The code above is unchanged on master as of 016542a.
• Windows 11 x64 and macOS.
• MindWork AI Studio uses one EdgeShard per local data source and calls flush() after every update().
• Our workaround until this is fixed: while a shard is not loaded, we delete all closed-* segments except the newest one, oldest first. We do this before EdgeShard::load and after optimize(), dropping the shard first. We would be glad to remove it again.