Repository navigation
fix(gateway): read uvicorn's WEB_CONCURRENCY in the remaining worker-count readers - #6253
Conversation
There was a problem hiding this comment.
[P2] Keep single-worker reload launches browser-compatible. With browser_navigate configured, GATEWAY_WORKERS unset, and WEB_CONCURRENCY=4, the supported foreground make dev/backend make dev now fails Gateway startup. Those launchers pass --reload; Uvicorn 0.46.0 selects its single-child reload supervisor instead of its multiprocess supervisor and ignores the worker count. This resolver nevertheless reports four workers, so browser_capability() rejects the configuration and ensure_browser_runtime_available() raises in lifespan initialization. Previously this environment resolved to one worker and started. Propagate the effective worker count/reload mode from the launchers before enforcing this guard, and cover reload with an exported WEB_CONCURRENCY>1 while retaining rejection for actual multi-worker launches.
There was a problem hiding this comment.
Confirmed: scripts/serve.sh --dev in the foreground and backend/Makefile dev both pass --reload with no --workers, so uvicorn supervises one serving child and an exported WEB_CONCURRENCY=4 never becomes the worker count. The refusal browser_capability() raises for that launch is a regression this commit introduced.
It shares the root cause of your [P2] on #6252: WEB_CONCURRENCY is only the effective count when the launcher omits --workers, and the worker process cannot see that flag. Verified at 9eb01b37 — compose always passes --workers ${GATEWAY_WORKERS:-1}; the two reload launchers force a single child; serve.sh --prod and --dev --daemon, backend/Makefile gateway and backend/Dockerfile CMD are the only paths where uvicorn's fallback applies.
I'd rather not duplicate the inference in a second _worker_count_from_env() here: fold these readers into the single GATEWAY_WORKERS-only helper from #6252 and let the no---workers launchers export the count uvicorn will use (GATEWAY_WORKERS=${GATEWAY_WORKERS:-${WEB_CONCURRENCY:-1}}). Reload with WEB_CONCURRENCY>1 would then resolve to one worker and start, while genuinely multi-process launches keep refusing. I'd add the reload regression here and keep the dedupe/session consistency tests.
Waiting on which PR should carry the launcher edits before pushing.
willem-bd
left a comment
There was a problem hiding this comment.
Reviewed at 5db347c. The core change is sound: both remaining readers now resolve the effective worker count the way uvicorn does, blank-as-unset and the inert-on-unparsable choice are reasonable, precedence (GATEWAY_WORKERS first) matches docker-compose, and explicit-count callers (deps._enforce_postgres_for_multi_worker) keep the documented-knob message as the new tests pin. The test coverage is thorough and the red/green verification is convincing. Three follow-ups below, none blocking; the confirmed reload-launcher regression from the earlier P2 (launcher must export the effective count, e.g. GATEWAY_WORKERS=${GATEWAY_WORKERS:-${WEB_CONCURRENCY:-1}}) is still the open item the author is tracking.
| with pytest.raises(RuntimeError, match="process-local"): | ||
| manager.get_session("thread-a") | ||
|
|
||
| async def test_get_session_rejects_worker_count_from_the_uvicorn_fallback(self, monkeypatch): |
There was a problem hiding this comment.
[test robustness] Reading a second, commonly-ambient variable makes the untouched success-path tests environment-sensitive: with WEB_CONCURRENCY>1 exported (a common uvicorn/gunicorn/PaaS convention) and GATEWAY_WORKERS unset, get_session() now raises for every test that does not control the env — test_get_session_is_per_thread_and_cached, the LRU/eviction block (~lines 800-880), and the runtime guard tests — plus the no-warning assertions in test_inbound_dedupe.py. The two multi-worker tests you added do delenv correctly; the rest of the module predates the second spelling. Consider a module-level autouse fixture that clears both GATEWAY_WORKERS and WEB_CONCURRENCY so the suite is independent of the invoking shell.
There was a problem hiding this comment.
Reproduced before touching anything, at the previous head 5db347c:
| condition | result |
|---|---|
tests/test_browser_automation.py, WEB_CONCURRENCY=4 exported, GATEWAY_WORKERS unset |
10 failed, 50 passed, 1 skipped |
| same module, both unset (control) | 60 passed, 1 skipped |
The failure is the one you predicted, raised from session.py through get_session() -> ensure_browser_worker_compatibility(): RuntimeError: WEB_CONCURRENCY=4 cannot enable agentic browser tools ....
Two corrections on scope, so the record is accurate rather than flattering to my fix:
tests/test_inbound_dedupe.pydid not fail in my run:20 passedunder the same ambientWEB_CONCURRENCY=4. Its factory tests already callmonkeypatch.delenv/setenvfor both spellings themselves. I added the fixture there anyway for the reason you give -- its green state was inherited from the invoking shell rather than asserted -- but I did not observe a failure there and would not want the fix justified by one.- The sensitivity was wider than the two modules in this diff.
tests/test_features_router.py::test_features_reports_browser_control_enabled_when_configured_and_runtime_availablealso broke, becausebrowser_capability.py:44callsbrowser_multi_worker_error()and now sees a refusal it did not previously compute. That one is a genuine regression introduced by this PR, not a pre-existing latent issue. Fixed in 864dce9 with the same fixture.
And it reaches #6252's file too: at that PR's previous head, with WEB_CONCURRENCY=4 exported and no fixture, the pre-existing test_gate_treats_invalid_env_as_single_worker fails with a SystemExit from the SQLite refusal (1 failed, 33 passed). So the fixture is now in test_multi_worker_postgres_gate.py as well.
After the fixes:
| condition | result |
|---|---|
tests/test_browser_automation.py, clean env |
62 passed, 1 skipped |
same, WEB_CONCURRENCY=4 |
62 passed, 1 skipped |
same, GATEWAY_WORKERS=8 |
62 passed, 1 skipped |
tests/test_inbound_dedupe.py, WEB_CONCURRENCY=4 |
21 passed |
| both modules together | 83 passed, 1 skipped |
| seven-module sweep (browser, dedupe, gate, channels, gateway config freshness, features router, agent storage) -- clean | 563 passed, 1 skipped, 2 errors |
same sweep, WEB_CONCURRENCY=4 |
563 passed, 1 skipped, 2 errors |
same sweep, GATEWAY_WORKERS=8 |
563 passed, 1 skipped, 2 errors |
Identical counts in all three conditions is the point of the fixture. The 2 errors are unrelated to worker count and reproduce with a clean environment: two TestDiscordChannel parametrised fixtures raise ValueError: the environment variable is longer than 32767 characters, which is this Windows host's limit on the long-CJK payload in the test ID, not a code problem.
ruff format --check reports all five touched files already formatted and ruff check passes on them.
Disclosure: prepared with Qoder (AI coding agent). Every number above was measured locally by that agent on this machine (Windows, Python 3.12.11) and is reproducible from the commands described.
| _DEFAULT_IDLE_TIMEOUT_S = 30 * 60.0 | ||
|
|
||
|
|
||
| # ``GATEWAY_WORKERS`` is the documented knob and the one docker-compose forwards as |
There was a problem hiding this comment.
[docs] Per the repo's documentation-update policy, the operator-facing text that says only "keep GATEWAY_WORKERS=1" is now incomplete: README.md (~line 2029), backend/docs/CONFIGURATION.md (~line 519), the config.example.yaml browser-group comment (~line 1132), and scripts/AGENTS.md all predate this change. After it, a deployment using the shipped Dockerfile/serve.sh launches (no --workers) is refused browser tools and warned on dedupe_storage=memory by WEB_CONCURRENCY alone, so the docs should say the gateway must be single-worker under either spelling (GATEWAY_WORKERS=1 and WEB_CONCURRENCY unset/1 when uvicorn reads it).
There was a problem hiding this comment.
All four locations were stale and are updated in eb0f6cf, phrased as you suggest -- the requirement is one Gateway worker process under either spelling:
README.md:2029-- browser sessions are process-local; keep the Gateway to one worker process, which meansGATEWAY_WORKERS=1, and alsoWEB_CONCURRENCYunset or1on the launches that pass uvicorn no worker count at all (backend/Dockerfile,scripts/serve.sh).backend/docs/CONFIGURATION.md:519-- same requirement inside the existing multi-worker paragraph, next to the Postgres / heartbeat /run_events.backend: dbconstraints.config.example.yaml:1131-1134-- thebrowsertool-group comment, now four lines covering both spellings.scripts/AGENTS.md:309-314-- the Gateway startup safety gate note.
I deliberately kept these edits to the browser-tool requirement and did not rewrite the surrounding GATEWAY_WORKERS > 1 guidance (Postgres backend, run-ownership heartbeats, run_events.backend: db): that guidance describes compose's --workers ${GATEWAY_WORKERS:-1} path, which is #6252's subject, and the reshape there is still undecided. If the launcher-propagation answer changes what operators are supposed to set, these four passages need a second pass -- flagging that rather than pre-empting it.
One consequence worth calling out for the docs, since it is not obvious from the diff: after this PR a WEB_CONCURRENCY=4 deployment on the Dockerfile/serve.sh launch paths is refused browser tools at startup, reports browser_control.enabled=false on /api/features, and is warned about dedupe_storage=memory, where before it got none of those. That is the intent, but it is a behaviour change for anyone who exports WEB_CONCURRENCY for another reason.
Also, since the docs policy note may be read as "the tests are the only contract": the features-router test that asserted browser_control.enabled=true had to be isolated from the shell's worker count for exactly this reason (864dce9).
Disclosure: prepared with Qoder (AI coding agent). Every number above was measured locally by that agent on this machine (Windows, Python 3.12.11) and is reproducible from the commands described.
There was a problem hiding this comment.
[nit] The two cross-references in the new comments are no longer accurate. This docstring says the helper mirrors deps._enforce_postgres_for_multi_worker, but deps reads only GATEWAY_WORKERS and resolves blanks differently ("" -> ValueError -> 1, " " -> int(" ") -> 0), while this helper treats both as unset and falls through to WEB_CONCURRENCY. Likewise "the way app/gateway/routers/channel_connections.py already does" (also in session.py's copy of the comment) overstates equivalence: that reader fail-closes — blank-but-present resolves to 0 and any count != 1 gets a 503, unparsable becomes 0 — whereas this helper fail-opens both (fallback / 1). The different strictness is defensible per reader, but the comment text should not claim parity, or the next reader will assume one shared semantic. Worth rewording while deciding where the single shared resolver from the #6252 discussion lands.
There was a problem hiding this comment.
Reworded in eb0f6cf. The docstring no longer claims this mirrors deps._enforce_postgres_for_multi_worker; it says this is a second reader rather than an import, and that its strictness differs from channel_connections.py on purpose. session.py's copy of the same comment now spells out the three-way difference instead of gesturing at equivalence: that route resolves blank-but-present and unparsable input to 0 and returns a 503 for any count != 1, while these readers fall through and stay inert.
One factual correction on the parse example, because it changes how the gap reads. int() does not turn whitespace into 0 -- measured on the same interpreter:
'' -> ValueError
' ' -> ValueError
' 2 ' -> 2
'abc' -> ValueError
So in deps the chain is GATEWAY_WORKERS=" " -> int(" ") raises ValueError -> the reader resolves to 1, not 0. The 0 you describe is real, but it comes from channel_connections.py:772-774, whose except ValueError: workers = 0 then trips if workers != 1 -> 503. Blank-means-unset is genuinely shared across all three readers (int('') raises, so the explicit raw.strip() guard is what implements it); what separates them is the unparsable branch, and channel_connections is the fail-closed one. The comment text now reflects that split, which I think is the distinction you wanted preserved rather than flattened.
Agree with the closing point -- different strictness per reader is defensible, the comment should not claim one shared semantic. Whether this stays three readers or becomes one shared resolver depends on the launcher-propagation decision in the #6252 [P2] thread, which is still open.
Disclosure: prepared with Qoder (AI coding agent). Every number above was measured locally by that agent on this machine (Windows, Python 3.12.11) and is reproducible from the commands described.
The browser refusal and the dedupe warning both took the count but printed GATEWAY_WORKERS, so on the launches that pass uvicorn no --workers the advice silences the gate while the extra processes keep starting. Resolve which variable set the count and print that, and let an unparsable GATEWAY_WORKERS fall through to WEB_CONCURRENCY instead of reporting a single worker. Also stop the comments from claiming the readers are equivalent: they share the two spellings and the blank-means-unset rule, but channel_connections.py turns an unparsable value into 0 and refuses while these stay inert.
The browser tool group and the inbound dedupe warning now refuse or warn on WEB_CONCURRENCY as well, so guidance that names only GATEWAY_WORKERS=1 is incomplete for the shipped Dockerfile and serve.sh launches, where uvicorn takes the process count from WEB_CONCURRENCY.
…'s worker count browser_capability() asks for the Gateway worker count, which this change makes readable from WEB_CONCURRENCY, so an exported count in the invoking shell turned test_features_reports_browser_control_enabled_when_configured_and_runtime_available into a failure that has nothing to do with the router.

Related to #6251 (already covered for the postgres gate by #6252). This PR does not close it — merging either one alone would leave the other readers unfixed.
Why
Trigger: while reading the multi-worker guards I found that the gateway worker count has two spellings in this repo.
GATEWAY_WORKERSis the documented knob and the onedocker/docker-compose.yamlforwards as--workers, butbackend/Dockerfileandscripts/serve.shstart uvicorn with no worker count at all — uvicorn then takes the count fromWEB_CONCURRENCY.app/gateway/routers/channel_connections.pyalready reads both; two other readers only readGATEWAY_WORKERS.Pain being addressed: in the
WEB_CONCURRENCYdeployment (the one the shipped Dockerfile produces) both guards stay inert:browser_multi_worker_error()is the fail-closed gate that refuses agentic browser tools because Chromium sessions are process-local and uvicorn gives no thread affinity. The runtime paths call it with no argument (app/gateway/browser_capability.py, andensure_browser_worker_compatibility()fromBrowserSessionManager.get_session), so withWEB_CONCURRENCY=4it resolvesGATEWAY_WORKERSas unset, returns 1, and multi-pod deployments enable process-local browser sessions anyway — sessions then land on whichever replica happens to handle the request, and a follow-up action on the samethread_idtargets a browser that is not in that process._gateway_workers()inapp/channels/dedupe_store.pydecides whetherdedupe_storage=memoryis safe. WithWEB_CONCURRENCY=4it reports single-worker, so the "inbound webhook dedupe is per-pod" warning never fires and a multi-worker pod silently believes it shares dedupe state.What changed
WEB_CONCURRENCY=4 cannot enable ... Set WEB_CONCURRENCY=1) instead of telling an operator to set a knob they never used.dedupe_storage=memorymulti-worker warning now also fires forWEB_CONCURRENCY, and names the variable the operator actually set.GATEWAY_WORKERSkeeps precedence when both are set, and a blank/whitespace value still counts as unset. Callers that pass the worker count explicitly are unchanged, byte for byte.Surface area
frontend/backend/applanggraph.json, or prompt changedocker/or sandboxed executionskills/backend/pyproject.tomlorfrontend/package.jsonThe two readers live in different layers (
backend/app/channels/and theharnesspackage, which never importsapp.*), so each keeps its own small resolver rather than a cross-layer import — same as the existingchannel_connections.pyreader.Bug fix verification
backend/tests/test_browser_automation.py::TestSessionManager::test_get_session_rejects_worker_count_from_the_uvicorn_fallbackbackend/tests/test_browser_automation.py::test_worker_count_resolution_reads_both_spellings(6 cases),::test_worker_count_error_names_the_variable_that_set_the_count,::test_worker_count_error_keeps_naming_the_documented_knob_when_passed_inbackend/tests/test_inbound_dedupe.py::test_factory_warns_under_the_uvicorn_worker_fallback,::test_factory_prefers_the_documented_knob_when_both_are_set,::test_factory_treats_a_blank_gateway_workers_as_unsetmainand green on this branch? yes — measured on this machine: 10 failed / 70 passed / 1 skipped with the tests checked out onmain, and 132 passed / 5 skipped on this branch acrosstest_inbound_dedupe.py,test_browser_automation.py,test_multi_pod_inbound_dedupe.py,test_browser_router.py,test_browser_egress.py.Validation
The 2 errors are pre-existing and unrelated to this diff:
tests/test_channels.py::TestDiscordChannel::test_split_text_preserves_content_within_discord_limitfails at setup/teardown on Windows withValueError: the environment variable is longer than 32767 characters(a unicode-escaped parametrized id exceeds the env limit). I reproduced them with this branch's four files reverted tomain.AI assistance
Tool(s) used: Qoder
How you used it: AI drafted the impl and the tests from a spec and ran the local RED/GREEN measurements; the diff is small, self-contained and reviewed line by line before pushing.
(Left unticked deliberately: this PR was opened by an agent on the account owner's standing instruction, so nobody has signed the "I read every line" claim as a human. Happy to tick it once the owner has read the diff.)