fix(gateway): read uvicorn's WEB_CONCURRENCY in the remaining worker-count readers by Lesereingrape · Pull Request #6253 · bytedance/deer-flow · GitHub
Skip to content

fix(gateway): read uvicorn's WEB_CONCURRENCY in the remaining worker-count readers - #6253

Merged
WillemJiang merged 4 commits into
bytedance:mainfrom
Lesereingrape:fix/worker-count-readers
Oct 4, 2026
Merged

WillemJiang merged 4 commits into
bytedance:mainfrom
Lesereingrape:fix/worker-count-readers

Conversation

@Lesereingrape

Copy link
Copy Markdown
Contributor

Related to #6251 (already covered for the postgres gate by #6252). This PR does not close it — merging either one alone would leave the other readers unfixed.

Why

Trigger: while reading the multi-worker guards I found that the gateway worker count has two spellings in this repo. GATEWAY_WORKERS is the documented knob and the one docker/docker-compose.yaml forwards as --workers, but backend/Dockerfile and scripts/serve.sh start uvicorn with no worker count at all — uvicorn then takes the count from WEB_CONCURRENCY. app/gateway/routers/channel_connections.py already reads both; two other readers only read GATEWAY_WORKERS.

Pain being addressed: in the WEB_CONCURRENCY deployment (the one the shipped Dockerfile produces) both guards stay inert:

  • browser_multi_worker_error() is the fail-closed gate that refuses agentic browser tools because Chromium sessions are process-local and uvicorn gives no thread affinity. The runtime paths call it with no argument (app/gateway/browser_capability.py, and ensure_browser_worker_compatibility() from BrowserSessionManager.get_session), so with WEB_CONCURRENCY=4 it resolves GATEWAY_WORKERS as unset, returns 1, and multi-pod deployments enable process-local browser sessions anyway — sessions then land on whichever replica happens to handle the request, and a follow-up action on the same thread_id targets a browser that is not in that process.
  • _gateway_workers() in app/channels/dedupe_store.py decides whether dedupe_storage=memory is safe. With WEB_CONCURRENCY=4 it reports single-worker, so the "inbound webhook dedupe is per-pod" warning never fires and a multi-worker pod silently believes it shares dedupe state.

What changed

  • Enabling browser tools in a multi-worker gateway now fails closed regardless of which variable set the count, and the error names that variable (WEB_CONCURRENCY=4 cannot enable ... Set WEB_CONCURRENCY=1) instead of telling an operator to set a knob they never used.
  • The dedupe_storage=memory multi-worker warning now also fires for WEB_CONCURRENCY, and names the variable the operator actually set.
  • GATEWAY_WORKERS keeps precedence when both are set, and a blank/whitespace value still counts as unset. Callers that pass the worker count explicitly are unchanged, byte for byte.
  • An unparsable value stays inert (count 1), matching uvicorn, which rejects it itself before serving.

Surface area

  • Frontend UI — page / component / setting / interaction under frontend/
  • Backend API — endpoint / SSE event / request-response shape under backend/app
  • Agents / LangGraph — agent node, graph wiring, langgraph.json, or prompt change
  • Sandbox — docker/ or sandboxed execution
  • Skills — change under skills/
  • Dependencies — new/upgraded entry in backend/pyproject.toml or frontend/package.json
  • Default behavior change — changes existing behavior without the user opting in (default model, default setting, data shape)
  • Docs / tests / CI only — no runtime behavior change

The two readers live in different layers (backend/app/channels/ and the harness package, which never imports app.*), so each keeps its own small resolver rather than a cross-layer import — same as the existing channel_connections.py reader.

Bug fix verification

  • Test paths that reproduce the bug:
    • backend/tests/test_browser_automation.py::TestSessionManager::test_get_session_rejects_worker_count_from_the_uvicorn_fallback
    • backend/tests/test_browser_automation.py::test_worker_count_resolution_reads_both_spellings (6 cases), ::test_worker_count_error_names_the_variable_that_set_the_count, ::test_worker_count_error_keeps_naming_the_documented_knob_when_passed_in
    • backend/tests/test_inbound_dedupe.py::test_factory_warns_under_the_uvicorn_worker_fallback, ::test_factory_prefers_the_documented_knob_when_both_are_set, ::test_factory_treats_a_blank_gateway_workers_as_unset
  • Did it go red on main and green on this branch? yes — measured on this machine: 10 failed / 70 passed / 1 skipped with the tests checked out on main, and 132 passed / 5 skipped on this branch across test_inbound_dedupe.py, test_browser_automation.py, test_multi_pod_inbound_dedupe.py, test_browser_router.py, test_browser_egress.py.

Validation

cd backend
.venv/Scripts/python.exe -m pytest -q tests/test_inbound_dedupe.py tests/test_browser_automation.py tests/test_multi_pod_inbound_dedupe.py tests/test_browser_router.py tests/test_browser_egress.py
# 132 passed, 5 skipped
.venv/Scripts/python.exe -m pytest -q tests/test_multi_worker_postgres_gate.py tests/test_channels.py tests/test_channel_intake_backpressure.py tests/test_inbound_dedupe.py tests/test_multi_pod_inbound_dedupe.py
# 477 passed, 3 skipped, 2 errors
ruff check .          # All checks passed
ruff format --check . # 4 files already formatted

The 2 errors are pre-existing and unrelated to this diff: tests/test_channels.py::TestDiscordChannel::test_split_text_preserves_content_within_discord_limit fails at setup/teardown on Windows with ValueError: the environment variable is longer than 32767 characters (a unicode-escaped parametrized id exceeds the env limit). I reproduced them with this branch's four files reverted to main.

AI assistance

Tool(s) used: Qoder

How you used it: AI drafted the impl and the tests from a spec and ran the local RED/GREEN measurements; the diff is small, self-contained and reviewed line by line before pushing.

  • I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.

(Left unticked deliberately: this PR was opened by an agent on the account owner's standing instruction, so nobody has signed the "I read every line" claim as a human. Happy to tick it once the owner has read the diff.)

@github-actions github-actions Bot added area:backend Gateway / runtime / core backend under backend/ risk:medium Medium risk: regular code changes size/M PR changes 100-300 lines labels Oct 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep single-worker reload launches browser-compatible. With browser_navigate configured, GATEWAY_WORKERS unset, and WEB_CONCURRENCY=4, the supported foreground make dev/backend make dev now fails Gateway startup. Those launchers pass --reload; Uvicorn 0.46.0 selects its single-child reload supervisor instead of its multiprocess supervisor and ignores the worker count. This resolver nevertheless reports four workers, so browser_capability() rejects the configuration and ensure_browser_runtime_available() raises in lifespan initialization. Previously this environment resolved to one worker and started. Propagate the effective worker count/reload mode from the launchers before enforcing this guard, and cover reload with an exported WEB_CONCURRENCY>1 while retaining rejection for actual multi-worker launches.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed: scripts/serve.sh --dev in the foreground and backend/Makefile dev both pass --reload with no --workers, so uvicorn supervises one serving child and an exported WEB_CONCURRENCY=4 never becomes the worker count. The refusal browser_capability() raises for that launch is a regression this commit introduced.

It shares the root cause of your [P2] on #6252: WEB_CONCURRENCY is only the effective count when the launcher omits --workers, and the worker process cannot see that flag. Verified at 9eb01b37 — compose always passes --workers ${GATEWAY_WORKERS:-1}; the two reload launchers force a single child; serve.sh --prod and --dev --daemon, backend/Makefile gateway and backend/Dockerfile CMD are the only paths where uvicorn's fallback applies.

I'd rather not duplicate the inference in a second _worker_count_from_env() here: fold these readers into the single GATEWAY_WORKERS-only helper from #6252 and let the no---workers launchers export the count uvicorn will use (GATEWAY_WORKERS=${GATEWAY_WORKERS:-${WEB_CONCURRENCY:-1}}). Reload with WEB_CONCURRENCY>1 would then resolve to one worker and start, while genuinely multi-process launches keep refusing. I'd add the reload regression here and keep the dedupe/session consistency tests.

Waiting on which PR should carry the launcher edits before pushing.

@willem-bd willem-bd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 5db347c. The core change is sound: both remaining readers now resolve the effective worker count the way uvicorn does, blank-as-unset and the inert-on-unparsable choice are reasonable, precedence (GATEWAY_WORKERS first) matches docker-compose, and explicit-count callers (deps._enforce_postgres_for_multi_worker) keep the documented-knob message as the new tests pin. The test coverage is thorough and the red/green verification is convincing. Three follow-ups below, none blocking; the confirmed reload-launcher regression from the earlier P2 (launcher must export the effective count, e.g. GATEWAY_WORKERS=${GATEWAY_WORKERS:-${WEB_CONCURRENCY:-1}}) is still the open item the author is tracking.

with pytest.raises(RuntimeError, match="process-local"):
manager.get_session("thread-a")

async def test_get_session_rejects_worker_count_from_the_uvicorn_fallback(self, monkeypatch):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[test robustness] Reading a second, commonly-ambient variable makes the untouched success-path tests environment-sensitive: with WEB_CONCURRENCY>1 exported (a common uvicorn/gunicorn/PaaS convention) and GATEWAY_WORKERS unset, get_session() now raises for every test that does not control the env — test_get_session_is_per_thread_and_cached, the LRU/eviction block (~lines 800-880), and the runtime guard tests — plus the no-warning assertions in test_inbound_dedupe.py. The two multi-worker tests you added do delenv correctly; the rest of the module predates the second spelling. Consider a module-level autouse fixture that clears both GATEWAY_WORKERS and WEB_CONCURRENCY so the suite is independent of the invoking shell.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced before touching anything, at the previous head 5db347c:

condition result
tests/test_browser_automation.py, WEB_CONCURRENCY=4 exported, GATEWAY_WORKERS unset 10 failed, 50 passed, 1 skipped
same module, both unset (control) 60 passed, 1 skipped

The failure is the one you predicted, raised from session.py through get_session() -> ensure_browser_worker_compatibility(): RuntimeError: WEB_CONCURRENCY=4 cannot enable agentic browser tools ....

Two corrections on scope, so the record is accurate rather than flattering to my fix:

  • tests/test_inbound_dedupe.py did not fail in my run: 20 passed under the same ambient WEB_CONCURRENCY=4. Its factory tests already call monkeypatch.delenv/setenv for both spellings themselves. I added the fixture there anyway for the reason you give -- its green state was inherited from the invoking shell rather than asserted -- but I did not observe a failure there and would not want the fix justified by one.
  • The sensitivity was wider than the two modules in this diff. tests/test_features_router.py::test_features_reports_browser_control_enabled_when_configured_and_runtime_available also broke, because browser_capability.py:44 calls browser_multi_worker_error() and now sees a refusal it did not previously compute. That one is a genuine regression introduced by this PR, not a pre-existing latent issue. Fixed in 864dce9 with the same fixture.

And it reaches #6252's file too: at that PR's previous head, with WEB_CONCURRENCY=4 exported and no fixture, the pre-existing test_gate_treats_invalid_env_as_single_worker fails with a SystemExit from the SQLite refusal (1 failed, 33 passed). So the fixture is now in test_multi_worker_postgres_gate.py as well.

After the fixes:

condition result
tests/test_browser_automation.py, clean env 62 passed, 1 skipped
same, WEB_CONCURRENCY=4 62 passed, 1 skipped
same, GATEWAY_WORKERS=8 62 passed, 1 skipped
tests/test_inbound_dedupe.py, WEB_CONCURRENCY=4 21 passed
both modules together 83 passed, 1 skipped
seven-module sweep (browser, dedupe, gate, channels, gateway config freshness, features router, agent storage) -- clean 563 passed, 1 skipped, 2 errors
same sweep, WEB_CONCURRENCY=4 563 passed, 1 skipped, 2 errors
same sweep, GATEWAY_WORKERS=8 563 passed, 1 skipped, 2 errors

Identical counts in all three conditions is the point of the fixture. The 2 errors are unrelated to worker count and reproduce with a clean environment: two TestDiscordChannel parametrised fixtures raise ValueError: the environment variable is longer than 32767 characters, which is this Windows host's limit on the long-CJK payload in the test ID, not a code problem.

ruff format --check reports all five touched files already formatted and ruff check passes on them.


Disclosure: prepared with Qoder (AI coding agent). Every number above was measured locally by that agent on this machine (Windows, Python 3.12.11) and is reproducible from the commands described.

_DEFAULT_IDLE_TIMEOUT_S = 30 * 60.0


# ``GATEWAY_WORKERS`` is the documented knob and the one docker-compose forwards as

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[docs] Per the repo's documentation-update policy, the operator-facing text that says only "keep GATEWAY_WORKERS=1" is now incomplete: README.md (~line 2029), backend/docs/CONFIGURATION.md (~line 519), the config.example.yaml browser-group comment (~line 1132), and scripts/AGENTS.md all predate this change. After it, a deployment using the shipped Dockerfile/serve.sh launches (no --workers) is refused browser tools and warned on dedupe_storage=memory by WEB_CONCURRENCY alone, so the docs should say the gateway must be single-worker under either spelling (GATEWAY_WORKERS=1 and WEB_CONCURRENCY unset/1 when uvicorn reads it).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All four locations were stale and are updated in eb0f6cf, phrased as you suggest -- the requirement is one Gateway worker process under either spelling:

  • README.md:2029 -- browser sessions are process-local; keep the Gateway to one worker process, which means GATEWAY_WORKERS=1, and also WEB_CONCURRENCY unset or 1 on the launches that pass uvicorn no worker count at all (backend/Dockerfile, scripts/serve.sh).
  • backend/docs/CONFIGURATION.md:519 -- same requirement inside the existing multi-worker paragraph, next to the Postgres / heartbeat / run_events.backend: db constraints.
  • config.example.yaml:1131-1134 -- the browser tool-group comment, now four lines covering both spellings.
  • scripts/AGENTS.md:309-314 -- the Gateway startup safety gate note.

I deliberately kept these edits to the browser-tool requirement and did not rewrite the surrounding GATEWAY_WORKERS > 1 guidance (Postgres backend, run-ownership heartbeats, run_events.backend: db): that guidance describes compose's --workers ${GATEWAY_WORKERS:-1} path, which is #6252's subject, and the reshape there is still undecided. If the launcher-propagation answer changes what operators are supposed to set, these four passages need a second pass -- flagging that rather than pre-empting it.

One consequence worth calling out for the docs, since it is not obvious from the diff: after this PR a WEB_CONCURRENCY=4 deployment on the Dockerfile/serve.sh launch paths is refused browser tools at startup, reports browser_control.enabled=false on /api/features, and is warned about dedupe_storage=memory, where before it got none of those. That is the intent, but it is a behaviour change for anyone who exports WEB_CONCURRENCY for another reason.

Also, since the docs policy note may be read as "the tests are the only contract": the features-router test that asserted browser_control.enabled=true had to be isolated from the shell's worker count for exactly this reason (864dce9).


Disclosure: prepared with Qoder (AI coding agent). Every number above was measured locally by that agent on this machine (Windows, Python 3.12.11) and is reproducible from the commands described.

Comment thread backend/app/channels/dedupe_store.py Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] The two cross-references in the new comments are no longer accurate. This docstring says the helper mirrors deps._enforce_postgres_for_multi_worker, but deps reads only GATEWAY_WORKERS and resolves blanks differently ("" -> ValueError -> 1, " " -> int(" ") -> 0), while this helper treats both as unset and falls through to WEB_CONCURRENCY. Likewise "the way app/gateway/routers/channel_connections.py already does" (also in session.py's copy of the comment) overstates equivalence: that reader fail-closes — blank-but-present resolves to 0 and any count != 1 gets a 503, unparsable becomes 0 — whereas this helper fail-opens both (fallback / 1). The different strictness is defensible per reader, but the comment text should not claim parity, or the next reader will assume one shared semantic. Worth rewording while deciding where the single shared resolver from the #6252 discussion lands.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworded in eb0f6cf. The docstring no longer claims this mirrors deps._enforce_postgres_for_multi_worker; it says this is a second reader rather than an import, and that its strictness differs from channel_connections.py on purpose. session.py's copy of the same comment now spells out the three-way difference instead of gesturing at equivalence: that route resolves blank-but-present and unparsable input to 0 and returns a 503 for any count != 1, while these readers fall through and stay inert.

One factual correction on the parse example, because it changes how the gap reads. int() does not turn whitespace into 0 -- measured on the same interpreter:

''      -> ValueError
'   '   -> ValueError
'  2  ' -> 2
'abc'   -> ValueError

So in deps the chain is GATEWAY_WORKERS=" " -> int(" ") raises ValueError -> the reader resolves to 1, not 0. The 0 you describe is real, but it comes from channel_connections.py:772-774, whose except ValueError: workers = 0 then trips if workers != 1 -> 503. Blank-means-unset is genuinely shared across all three readers (int('') raises, so the explicit raw.strip() guard is what implements it); what separates them is the unparsable branch, and channel_connections is the fail-closed one. The comment text now reflects that split, which I think is the distinction you wanted preserved rather than flattened.

Agree with the closing point -- different strictness per reader is defensible, the comment should not claim one shared semantic. Whether this stays three readers or becomes one shared resolver depends on the launcher-propagation decision in the #6252 [P2] thread, which is still open.


Disclosure: prepared with Qoder (AI coding agent). Every number above was measured locally by that agent on this machine (Windows, Python 3.12.11) and is reproducible from the commands described.

The browser refusal and the dedupe warning both took the count but printed
GATEWAY_WORKERS, so on the launches that pass uvicorn no --workers the advice
silences the gate while the extra processes keep starting. Resolve which
variable set the count and print that, and let an unparsable GATEWAY_WORKERS
fall through to WEB_CONCURRENCY instead of reporting a single worker.

Also stop the comments from claiming the readers are equivalent: they share
the two spellings and the blank-means-unset rule, but channel_connections.py
turns an unparsable value into 0 and refuses while these stay inert.
The browser tool group and the inbound dedupe warning now refuse or warn on
WEB_CONCURRENCY as well, so guidance that names only GATEWAY_WORKERS=1 is
incomplete for the shipped Dockerfile and serve.sh launches, where uvicorn
takes the process count from WEB_CONCURRENCY.
@github-actions github-actions Bot added area:ci GitHub Actions, CI config, repo tooling area:docs Documentation and Markdown only labels Oct 4, 2026
…'s worker count

browser_capability() asks for the Gateway worker count, which this change makes
readable from WEB_CONCURRENCY, so an exported count in the invoking shell turned
test_features_reports_browser_control_enabled_when_configured_and_runtime_available
into a failure that has nothing to do with the router.
@WillemJiang WillemJiang added this to the 2.2.0 milestone Oct 4, 2026
@WillemJiang
WillemJiang merged commit e7e5a5e into bytedance:main Oct 4, 2026
27 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:backend Gateway / runtime / core backend under backend/ area:ci GitHub Actions, CI config, repo tooling area:docs Documentation and Markdown only risk:medium Medium risk: regular code changes size/M PR changes 100-300 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants