Print what the server said when an inference smoke request 4xx's - #9202
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 250336cfa4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
Verify the response body reaches the diagnostic print
This assertion only checks that a printed expression contains code; a future handler can still call exc.read() but print only the status code and pass the suite, recreating the exact body-loss regression this guard is intended to prevent. Require the diagnostic print to reference detail (or otherwise prove that the value derived from read() reaches the log).
Useful? React with 👍 / 👎.
|
Three checks are red here and none of them is this PR. This branch changes three workflow YAMLs and adds one test file; all three failures reproduce on
Merging #9189 first clears the first two here. |
…ckage
Five tests guard themselves with `pytest.importorskip("playwright")` and then
import a harness that does `from playwright.sync_api import Page`. On the Repo
tests (CPU) runner the top-level name resolves as a namespace directory with no
sync_api inside it, so the guard passes and the import dies with
ImportError: cannot import name 'Page' from 'playwright.sync_api' (unknown location)
A skip condition reported as a failure, on every branch, for as long as that
runner stays that way. It is red on #9202 and #9213 too, neither of which touches
any of this.
One of the two files already said what the guard was really for: "importing a
harness pulls in playwright.sync_api". It now checks that.
…lent 30 minutes This step stalls. Three times in one day it sat in apt's download loop until the job's 30-minute timeout killed it, while the sibling shards finished the whole job in 4 to 9 minutes. Twice on #9202 and once on #9189, always the same step. The cost is out of proportion to the cause. GitHub scores a job timeout as "cancelled" rather than a failure, prints no reason, and skips every step after it, so the chat shard reported nothing about the chat surface for what both times turned out to be an infrastructure hiccup that cleared on a plain re-run of the same commit. A per-attempt `timeout` turns the stall into a failure instead of a silent wait, and the retry is what actually recovers. The healthy time is about 2 minutes, so 8 per attempt is 4x headroom and a merely slow mirror will not trip it. timeout-minutes bounds the pair in case `timeout` is outlived by an unkillable child. Same reasoning, and the same wording, as the bounded prime-hf step in studio-mac-ui-smoke.yml. Both install steps in this file, since ui-smoke and ui-indicator run the identical command. Left alone on mac and windows: neither passes --with-deps, so neither has the apt phase this is about, and neither has been observed to stall.
…t guard it (#9189) * Repair the two chat auto-load suites that #9173 left red on main #9173 added a vision-projector fallback to the auto-load success toast. Both of the suites that read that code broke on it, and both were green on the commit immediately before (54b6ca4). 72 tests, in two different shapes: 1. tests/studio/test_chat_autoload_failure_gate.py, 71 failures. The harness slices autoLoadSmallestModel verbatim out of chat-adapter.ts and runs it, so anything the slice references must exist in PREAMBLE. mmprojFallbackMessage did not, which is a bare ReferenceError inside the retry loop; the loop catches it and scores it as a failed load, so every scenario fails as a wrong-model assertion. The file's own guard caught this and named the symbol, which is what it was written for after #7699 did the same thing. Stubbed as a function of the reason rather than a copy of the real three-message record. The value only reaches `options.description`, and these scenarios assert on which model loaded, never on toast copy, so copying user-facing strings in here would give them a second home to drift from. 2. tests/studio/test_model_picker_contracts.py, 1 failure. It asserted the literal `description: cpuFallbackReason`, and the mmproj branch went in front of it. The property held; only the spelling moved. Its own comment records this happening once already, when the CPU-fallback branch first appeared, so it now pins the property: the description varies on both fallback reasons and still has an undefined arm for the ordinary path. Scoped to the description EXPRESSION, not the whole helper. `cpuFallbackReason` is also a parameter name in the signature above, so a substring test over the block stays green with the CPU branch deleted outright -- the first cut of this check did exactly that, and mutation caught it. Three mutations verified red: the description no longer driven by any fallback reason, the CPU branch dropped, and the mmproj branch dropped. 4223 passed, 4 skipped. * Say both fallbacks when both fire, not just the projector one Both load paths wrote the toast description as mmprojFallbackReason ? mmprojMessage : cpuFallbackReason ? cpuMessage : undefined so whenever both reasons are set the CPU-fallback sentence is dropped. The user is told "loaded without vision" and never told the model is running on the CPU, which reads as a deliberate, explained degradation rather than an unaccelerated session. The combination is reachable. On a CPU-fallback replay llama_cpp.py preserves _cpu_fallback_reason (it clears it only when not _replaying_cpu_fallback) and resets _mmproj_fallback_reason so the projector can fail again inside that same launch. A low VRAM machine whose Vulkan backend crashed is exactly where the projector then falls back too. loadFallbackNotice() in mmproj-fallback.ts is now the single composition of the title suffix, the description and the degraded flag, and both call sites delegate to it. CPU_FALLBACK_MESSAGE moves there as well, so the two paths cannot describe the same condition differently again. Tests: four combined-case cases in mmproj-fallback.test.ts, verified red against the shipped nested ternary. test_model_picker_contracts.py now asserts the call sites delegate and pass both reasons rather than matching the old inline ternary, and test_chat_autoload_failure_gate.py's stub mirrors the composition so a call site dropping a reason stays detectable. * Repin the CPU-fallback toast test to behaviour, and unbreak the queued-capabilities test Two frontend suites were red. auto-load-cpu-fallback-toast.test.ts matched the warn-vs-success choice and the message text as substrings of showAutoLoadSuccess. Both moved into loadFallbackNotice, which is the single definition the explicit-load path now shares, so the match went stale. Matching the inline form again would go red on a refactor that changes nothing a user can see, and would stay green if only one of the two load paths kept the behaviour. It now calls loadFallbackNotice and asserts the verdict, and separately asserts the call site delegates to it. queued-model-capabilities.test.ts was red on main before this branch. #9173 added `import { isTextOnlyMmprojFallback } from "./mmproj-fallback"` to image-input-support.ts, which the test imports statically. Extensionless is the right form -- 2314 of the 2367 relative imports under src/ are written that way, and vite and tsconfig's "bundler" mode resolve them -- but the bare node loader does not, and a static import resolves before any registration can run. The test now registers the bundler resolver and imports dynamically, which is what mmproj-fallback.test.ts already does for the same module. Both files reformatted by biome; regex literals hoisted out of the test bodies for useTopLevelRegex. Full frontend suite: 3734 pass, 0 fail. typecheck clean. The 4 biome errors in chat-adapter.ts and use-chat-model-runtime.ts are byte-identical on origin/main. * Order the split-axis abort against the mmproj strip, not against its argument test_tensor_split_abort_raises_early_to_layer_fallback has been red on main since #9173, which renamed the text-only strip's argument from _last_spawn_cmd to _vision_gpu_cmd. That rename is right: the strip should read the vision GPU command rather than whatever was spawned last, and #9173 refreshes _last_spawn_cmd from the result immediately after. The test was pinned to the old argument name, so a rename with no behavioural content took it down. The failure also misreported itself. `assert raise_idx < src.find(needle)` reads as an ordering check but is two claims at once, and when the landmark is gone it fails with "assert 249423 < -1" -- which says the ordering broke, when what happened is that the landmark moved. Each landmark is now required to exist before it is ordered, and says so. The strip is matched on the call rather than on what is passed to it. What this test is about is that the abort raises BEFORE the projector is discarded (#6659); which command the strip reads from is that code's own business. Checked both ways: removing the strip call from load_model goes red with a message naming the missing landmark, and renaming the argument again stays green. * Name the endpoints when the heavy-thread harness catches a stray request The harness records every /api/ URL issued during a measured action, then keeps only the count, so the failure reads let 2 /api/ requests reach the network during the measured actions and stops there. It says an interaction paid for a round trip without saying which one, and the reader has to bisect the frontend to learn what the harness already knew and discarded. It now reports the endpoints. Deduplicated and capped at eight, because the case this instrument exists to catch is a request issued once per message, which would otherwise print hundreds of copies of one line. This is why it surfaced now: the step has not run on main since #9173, whose unit test break fails earlier in the same job and short-circuits it. It was last green at 54b6ca4. With the unit tests repaired on this branch the job reaches the step again, and the first thing it needed to say was the one thing it did not. * Skip the playwright harness tests on the module they need, not the package Five tests guard themselves with `pytest.importorskip("playwright")` and then import a harness that does `from playwright.sync_api import Page`. On the Repo tests (CPU) runner the top-level name resolves as a namespace directory with no sync_api inside it, so the guard passes and the import dies with ImportError: cannot import name 'Page' from 'playwright.sync_api' (unknown location) A skip condition reported as a failure, on every branch, for as long as that runner stays that way. It is red on #9202 and #9213 too, neither of which touches any of this. One of the two files already said what the guard was really for: "importing a harness pulls in playwright.sync_api". It now checks that. * Stop the fork-count store asking the server about threads it has never seen Two fixes, both found by the heavy-thread smoke once it could name what it caught. The smoke reported "let 2 /api/ requests reach the network during the measured actions" and, with the endpoints now printed, they were POST /api/chat/threads/__LOCALID_lsQbsDZ/forks A `__LOCALID_` thread has no server record, so that request can only 404, and getThreadForkCounts already maps 404 to the empty map the entry starts as. It is a round trip whose answer is known before it is sent. Not a rounding error. A new chat is in exactly that state, and this store refreshes on CHAT_HISTORY_UPDATED_EVENT, which fires once per streaming chunk, so the first reply in a new chat paid one useless request per debounce window for as long as it streamed. #8992 added the store to stop the chat getting slower as a thread fills; excluding threads the server has never seen is the same intent. thread-ids.ts already had the predicate. Two tests: a local thread must not fetch on subscribe or on a burst of history events, and a saved thread on screen beside it must still refresh -- the guard has to be per thread, not a global off switch. They import the real isAssistantLocalThreadId rather than restating the prefix, so the rule under test cannot drift from the app's. Both go red with the guard removed. Second fix, same job: the playwright skip guard. The previous commit moved it from "playwright" to "playwright.sync_api" and it still failed, because sync_api resolves as a namespace package on that runner too. Only the symbol the harnesses import distinguishes a usable install, so the guard now checks for Page the way the harness does. Verified both ways against a Page-less sync_api: it skips, and it still proceeds when Page is there. Frontend suite 3789 pass, typecheck clean, tests/studio 4237 pass. * Bound and retry the Playwright browser install so a stall is not a silent 30 minutes This step stalls. Three times in one day it sat in apt's download loop until the job's 30-minute timeout killed it, while the sibling shards finished the whole job in 4 to 9 minutes. Twice on #9202 and once on #9189, always the same step. The cost is out of proportion to the cause. GitHub scores a job timeout as "cancelled" rather than a failure, prints no reason, and skips every step after it, so the chat shard reported nothing about the chat surface for what both times turned out to be an infrastructure hiccup that cleared on a plain re-run of the same commit. A per-attempt `timeout` turns the stall into a failure instead of a silent wait, and the retry is what actually recovers. The healthy time is about 2 minutes, so 8 per attempt is 4x headroom and a merely slow mirror will not trip it. timeout-minutes bounds the pair in case `timeout` is outlived by an unkillable child. Same reasoning, and the same wording, as the bounded prime-hf step in studio-mac-ui-smoke.yml. Both install steps in this file, since ui-smoke and ui-indicator run the identical command. Left alone on mac and windows: neither passes --with-deps, so neither has the apt phase this is about, and neither has been observed to stall. * Make the Playwright install retry able to actually recover The bound added in the previous commit worked: the stall became an 8m37s step FAILURE with a complete log instead of a silent 30-minute cancellation, and the log named the cause on the first try. It also showed the retry could not work. playwright shells out to apt-get as root, so terminating the python parent leaves that child alive holding the lock, and attempt 2 died two seconds later with E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 4578 (apt-get) A retry that cannot succeed is worse than no retry: it buries the real reason under a second, different failure. Attempt 2 now waits for the lock to clear, up to two minutes, and only then takes it -- the holder is our own orphan and the runner is a throwaway. Two smaller things the same log exposed. --kill-after was missing, so a process that ignores SIGTERM would have been waited on forever inside the step bound. And the warning said "did not finish within 8 minutes" about a two-second exit, which sends the next reader looking for a stall that never happened; it now separates timeout's 124/137 from playwright refusing outright, and prints the status. timeout-minutes 18 to 22 to cover two 8-minute attempts plus the lock wait, still inside the job's 30.
broken for three main runs, and the only thing CI printed was
urllib.error.HTTPError: HTTP Error 400: Bad Request
The response body carries llama-server's own explanation and it went out unread,
so the cause had to be reconstructed by hand from the workflow source. #9155
fixed the regression; this stops the next one costing the same dig.
The HTTPError branch of every request helper in the three inference smoke
workflows (nine sites) now prints the status, the reason and the body before
re-raising. The read itself is guarded, because a truncated or already-consumed
body must not replace the real status with a confusing one, and the original
error still propagates: reporting is not tolerating.
The tenth site, the tool-probe seed loop in studio-inference-smoke.yml, is a
caller rather than a helper. post_sse has already printed the body by the time
it re-raises there.
tests/studio/test_inference_smoke_http_diagnostics.py parses the Python actually
embedded in the workflows with ast rather than matching text, so a rewrite that
keeps the behaviour keeps passing. It also asserts every embedded probe parses at
all, which nothing else did: the heredocs are shell text inside YAML, invisible
to every linter in the repo. Seven mutations checked red (revert the change, drop
the re-raise, drop the print, drop the status code, unguard the read, remove the
handler, break the syntax).
Listed in workflow-trigger-lint because it reads workflow files, so the edit that
breaks it is workflow-only and no paths filter would collect it. Confirmed by
removing the line and watching test_workflow_guards_run_unfiltered name it.
for more information, see https://pre-commit.ci
d2c812b to
9b7761a
Compare

The Mac GGUF probe was red on three consecutive main runs after #8883. Everything CI printed about it was this:
llama-server's own explanation of the 400 is in the response body, and the body went out unread. Finding the cause meant pulling the raw job log, counting lines into a heredoc to work out which call had failed, and reasoning about the request from the workflow source. #9155 fixed the regression itself; this is so the next one does not cost the same dig.
These probes are the only place in CI where a real llama-server answers a real request, so their diagnostics are most of the value of a red run.
The change
The
HTTPErrorbranch of every request helper in the three inference smoke workflows (nine sites, identical) now prints the status, the reason and the body before re-raising:The read is guarded because a truncated or already-consumed body must not replace the real status with a confusing one, and the original error still propagates: reporting is not tolerating.
A tenth
except urllib.error.HTTPErrorexists, in the tool-probe seed loop ofstudio-inference-smoke.yml. It is a caller, not a helper.post_ssehas already printed the body by the time it re-raises there, so it is left alone.The guard
tests/studio/test_inference_smoke_http_diagnostics.pyextracts thepython - <<'PY'blocks and parses them withast, so it checks behaviour rather than text and a rewrite that keeps the reporting keeps passing. Three properties:urlopenhandlesHTTPErrorseparately. Without that it falls into theURLErrorbranch it subclasses and gets retried as a transport stall, spending the job's timeout budget re-asking a question the server already refused.try, prints it with the status code, and re-raises.Seven mutations confirmed red: revert the change, drop the re-raise, drop the print, drop the status code from the print, unguard the read, delete the handler, break the embedded syntax.
Listed in
workflow-trigger-lintbecause it reads workflow files, so the edit that breaks it is workflow-only and no paths filter would collect it. Confirmed by removing the line and watchingtest_workflow_guards_run_unfilteredname the module.Verification
pytest tests/studio/ -q -n 16: 71 failures, all of them the two chat auto-load modules that are red on main today and are fixed in #9189. No module touched by this branch fails.