Commit Qdrant to payload-partitioned multitenancy: drop per-collection shard keys, retire the in-process lock table, recover fencing and reclamation · Issue #1564 · MemMachine/MemMachine · GitHub
Skip to content

Commit Qdrant to payload-partitioned multitenancy: drop per-collection shard keys, retire the in-process lock table, recover fencing and reclamation #1564

Description

@edwinyyyu

Qdrant offers two ways to separate tenants, and they have opposite cost profiles. This issue tracks committing to the scalable one and finishing it, rather than leaving both half-supported.

Measured against Qdrant 1.19.0, one hundred tenants each way:

shard key per collection payload partitioning
admitting a tenant 450 ms, explicit API call no call at all
100 tenants 45.0 s 0.3 s including 10,000 points
segments 505 5, and still 5 after delete and reuse
deleting one tenant O(1) drop 9 ms filter-delete for 100 points
fencing of stale writes yes, Shard key "t1" not found none
cluster mode required, plus a bootstrap --uri not required

Payload partitioning is the only one that scales, because a tenant is a value rather than a physical structure. Everything below follows from choosing it.

1. Stop creating a shard key per logical collection

QdrantConf.is_distributed switches to custom sharding and gives every logical collection its own shard key:

sharding_method=models.ShardingMethod.CUSTOM if self._is_distributed else None
...
if self._is_distributed:
    await self._ensure_shard_key(entry.native_collection_name, entry.partition_key)

The cost is per logical collection. A deployment serving thousands of tenants — a tenant being a user, or a group of users — has at least that many logical collections and in practice more, since a tenant's memory is typically partitioned further. At a thousand logical collections that is ~7.5 minutes of shard-key creation and ~5,000 segments; at ten thousand, ~75 minutes and ~50,000 segments, before any data is written.

Qdrant's own guidance recommends against per-tenant physical structures: "Creating a separate collection for each tenant is rarely the most efficient approach. Each collection carries its own resource overhead."

The flag's docstring sells the O(1) delete, which is genuine, but it is paid for per tenant. Note the two mechanisms are layered correctly — the payload partition key is written and filtered in both modes, with the shard key passed additionally — so this is a policy problem, not a correctness one.

If a container ever needs splitting, the right lever is shard_number > 1 with automatic sharding: a provisioning knob, no custom sharding, no per-tenant cost.

2. Retire the per-process lock table

_name_locks serialises lifecycle operations within one process, which was the only arbitration available before a cross-process registry existed. It also leaks:

# Keyed by client so locks are garbage-collected when the client is.
_name_locks: ClassVar[
    WeakKeyDictionary[AsyncQdrantClient, defaultdict[tuple[str, str], asyncio.Lock]]
] = WeakKeyDictionary()

The outer mapping is weak, but the inner defaultdict gains a lock per (namespace, name) the process ever touches and never drops one, for the client's lifetime. Small individually, unbounded in the number of logical collections a process serves, and easy to miss because the comment reads as though the lifetime question is settled.

Once the registry arbitrates lifecycle across processes (#1524, #1525, delivered by #1526 and #1527), these locks are vestigial and should go rather than be bounded.

3. Recover what payload partitioning gives up

Choosing it means giving up the two things the shard-key path provided, both of which have to be built back:

Tiered multitenancy stays available

Promoting a single oversized tenant to its own shard key, while the long tail stays payload-partitioned, is Qdrant's third documented approach and remains open. sharding_method is immutable per collection — verified, creating a shard key on an auto-sharded collection fails with "Shard Key cannot be created with Auto sharding method" — but since registry entries pin their native collection, promotion later means provisioning a new custom-sharded collection and directing new collections to it, leaving existing ones untouched. Nothing here forecloses it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions