feat: limit concurrent chat agents with pooled admission - #27902
Conversation
|
@codex review |
Documentation CheckNew Documentation Needed
Note The auto-generated doc changes in this PR ( Automated review via Coder Agents |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 488948e1e1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
DanielleMaywood
left a comment
There was a problem hiding this comment.
Will perform a more thorough review of the backend tomorrow when I'm more awake but the frontend changes are fine with me
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1eac94c228
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8959eba6c9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 05375b8e7d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2b5a180a2b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 21e7706a55
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review
|
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Didn't find any major issues. Swish! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
8d05987 to
e7300dc
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e7300dcc2f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…rker Replace the capacity_queued_at column and its mark/clear machinery with state derived from ownership, status, and pool fullness. AgentAdmission shrinks to a single Admit method; a separate AgentCapacityPolicy exposes current limits for read-side derivation. Queue metrics and capacity events move to the OSS worker, driven by a per-replica in-memory map of refused chats. FIFO ordering now keys on updated_at alone.
Replica A can refuse a chat and publish the queued capacity event while replica B later admits it. B has no local refusal entry, so gating the clear event on the local capacity queue left the banner stale until the next refetch. On capped deployments, admission of a running chat now always publishes the clear. Also assert FIFO admission on the serial Admit call order recorded by fakeAdmission instead of task-start order, which goroutine scheduling can reorder, and document the requires_action resume overage bound.
Review feedback from cian: AgentAdmission and AgentCapacityPolicy always shipped as the same object, so replace them with a single AgentCapacityLimiter interface (Admit + Limits). Limits returns the caps plus a capped bool instead of a Capped field. A noop limiter replaces nil checks; chatd wires it whenever no factory is configured. Consolidate CountChatCapacityActiveByPool and CountChatCapacityUnownedByPool into CountChatCapacityByPool so admission and the metrics loop share one scan, and widen idx_chats_capacity_active to cover unowned chats so the merged query stays indexed.
…ents Codex review findings on 546e293: - An interrupting chat acquired by a replica that never refused it kept the queued banner because the clear only covered running acquisitions. Interrupting acquisitions now publish the clear too. - A delayed queued capacity event could arrive after another replica's clear and restore the banner. Acquisition bumps updated_at while refusals do not, so the frontend merge now applies capacity events only when the snapshot is not older than the cache. Also add the concurrent-agents docs section and point the queued banner's learn-more links at it.
Codex found the cap can flip to uncapped (license update) between one replica's refusal, which published queued=true, and another replica's admission, which then read capped=false and skipped the clear, leaving a stale banner. The clear condition now depends only on whether a real limiter is configured: the noop limiter never refuses, so no queued event can exist, while any real limiter may have refused while capped. No dynamic state can desync the clear from the queued event anymore.
…pshot Codex found the skipped-pool path published queued=true from a stale candidate row: another replica can acquire the chat between the batch query and the publish, and because the event payload reloaded the chat, it carried the acquisition's new updated_at and defeated the frontend freshness guard when delivered after the owner's clear. enterCapacityQueue now validates and publishes from one snapshot: it only queues chats that are still running, unarchived, and without a live owner heartbeat, and the event payload is that same snapshot, so a stale queued event either never publishes or carries pre-acquisition updated_at that loses to the clear. Crashed-owner takeovers still queue because their heartbeat is stale.
…kipped passes Codex found prune starvation: with a sustained full-pool backlog every acquisition pass ends on an all-skipped batch, which returned before pruneCapacityQueue could run, so an entry for a chat another replica acquired persisted indefinitely, suppressing the next episode's queued event and measuring wait time from the previous episode. An all-skipped pass now reconciles the local entries it did not see with one targeted FilterChatCapacityWaiting query and drops those that stopped being capacity-wait candidates. One bounded query per pass, instead of paging the global candidate stream, which under churn had no termination bound.
An all-skipped acquisition pass ended before reaching chats sorted behind the known queue heads, so a chat arriving on a saturated pool never got its queued event and open tabs kept showing plain running state. Reconcile now lists all waiting chats once per all-skipped pass, pruning departed entries and queueing arrivals in pools the pass proved full.
…races Two races found by review: Resuming a chat from waiting or error during the runner's abandon gap preserved worker_id/runner_id, so generation restarted on the retained runner without capacity admission while the slot release may have already admitted a queued chat. Idle-state resumes now clear ownership, routing the resume through worker acquisition and re-admission. A capacity clear published at acquisition could be rejected by the frontend freshness guard when a newer status event advanced the cached updated_at first, leaving the queued banner stuck on a generating chat. Capacity events now order against a per-chat capacity revision (the newest capacity event's updated_at) instead of the general guard.
…ity invalidation The ordering guards can reject a legitimate capacity event when delivery reorders against message sends that bump updated_at (for example a queued event loaded before but delivered after a send to the same still-unowned chat). The worker keeps its local queue entry and emits no replacement event, so the banner would stay wrong until an unrelated refetch. After merging a capacity event into the caches, invalidate the chat entity when the merged result does not reflect the event; the single-chat GET derives the authoritative queued state.
… pubsub nil check
…allocation feature
…ing repair override
2acf9f1 to
f56a4e2
Compare

Limits concurrent chat generation on capped deployments to 5 root chats and 10 delegated subagent chats. The pools are deployment-wide and independent, so delegated work can continue while root capacity is full.
The default caps live in AGPL code. Enterprise contributes only a licensing unlock, so unlicensed deployments stay capped and cannot fail open. Licensed deployments are uncapped while Agent Hours usage stays below an explicit hard limit. Deployments without a hard limit remain uncapped, and reaching the Agent Hours allocation only triggers warnings.
Admission happens before a worker takes chat ownership. Capped deployments serialize admission across replicas with a transaction-scoped advisory lock and derive active and queued state from current ownership plus fresh runner heartbeats, rather than persisted queue markers or per-replica state. The acquisition query returns a bounded, pool-interleaved candidate set instead of ranking the whole backlog; a migration replaces the acquisition index with a pool-aware one. Refused chats stay running but unowned, and interrupt requests bypass admission so users can stop queued or over-cap chats.
The single-chat API derives
queued_for_capacityfrom live pool state; list endpoints do not report it. The UI polls that value every 5 seconds while a chat is running and shows a callout when the chat is waiting for capacity.Updates the administrator documentation and deployment-wide Prometheus gauges for active and queued agents. Replica-level values must be aggregated with
max, notsum.