Print what the server said when an inference smoke request 4xx's by danielhanchen · Pull Request #9202 · unslothai/unsloth · GitHub
Skip to content

Print what the server said when an inference smoke request 4xx's - #9202

Merged
danielhanchen merged 3 commits into
mainfrom
smoke-http-detail
Aug 19, 2026
Merged

Print what the server said when an inference smoke request 4xx's#9202
danielhanchen merged 3 commits into
mainfrom
smoke-http-detail

Conversation

@danielhanchen

Copy link
Copy Markdown
Member

The Mac GGUF probe was red on three consecutive main runs after #8883. Everything CI printed about it was this:

urllib.error.HTTPError: HTTP Error 400: Bad Request

llama-server's own explanation of the 400 is in the response body, and the body went out unread. Finding the cause meant pulling the raw job log, counting lines into a heredoc to work out which call had failed, and reasoning about the request from the workflow source. #9155 fixed the regression itself; this is so the next one does not cost the same dig.

These probes are the only place in CI where a real llama-server answers a real request, so their diagnostics are most of the value of a red run.

The change

The HTTPError branch of every request helper in the three inference smoke workflows (nine sites, identical) now prints the status, the reason and the body before re-raising:

except urllib.error.HTTPError as exc:
    try:
        detail = exc.read().decode("utf-8", "replace")
    except Exception as read_exc:
        detail = f"<body unreadable: {read_exc!r}>"
    print(
        f"[http] {path} -> {exc.code} {exc.reason}: {detail[:4000]}",
        flush = True,
    )
    raise

The read is guarded because a truncated or already-consumed body must not replace the real status with a confusing one, and the original error still propagates: reporting is not tolerating.

A tenth except urllib.error.HTTPError exists, in the tool-probe seed loop of studio-inference-smoke.yml. It is a caller, not a helper. post_sse has already printed the body by the time it re-raises there, so it is left alone.

The guard

tests/studio/test_inference_smoke_http_diagnostics.py extracts the python - <<'PY' blocks and parses them with ast, so it checks behaviour rather than text and a rewrite that keeps the reporting keeps passing. Three properties:

  • every embedded probe parses as Python. Nothing else in the repo checks this: the heredocs are shell text inside YAML, invisible to actionlint, ruff and every YAML linter.
  • every helper that calls urlopen handles HTTPError separately. Without that it falls into the URLError branch it subclasses and gets retried as a transport stall, spending the job's timeout budget re-asking a question the server already refused.
  • that handler binds the error, reads the body under a try, prints it with the status code, and re-raises.

Seven mutations confirmed red: revert the change, drop the re-raise, drop the print, drop the status code from the print, unguard the read, delete the handler, break the embedded syntax.

Listed in workflow-trigger-lint because it reads workflow files, so the edit that breaks it is workflow-only and no paths filter would collect it. Confirmed by removing the line and watching test_workflow_guards_run_unfiltered name the module.

Verification

pytest tests/studio/ -q -n 16: 71 failures, all of them the two chat auto-load modules that are red on main today and are fixed in #9189. No module touched by this branch fails.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 250336cfa4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +200 to +201

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Verify the response body reaches the diagnostic print

This assertion only checks that a printed expression contains code; a future handler can still call exc.read() but print only the status code and pass the suite, recreating the exact body-loss regression this guard is intended to prevent. Require the diagnostic print to reference detail (or otherwise prove that the value derived from read() reaches the log).

Useful? React with 👍 / 👎.

@danielhanchen

Copy link
Copy Markdown
Member Author

Three checks are red here and none of them is this PR. This branch changes three workflow YAMLs and adds one test file; all three failures reproduce on origin/main unmodified.

Merging #9189 first clears the first two here.

danielhanchen added a commit that referenced this pull request Aug 18, 2026
…ckage

Five tests guard themselves with `pytest.importorskip("playwright")` and then
import a harness that does `from playwright.sync_api import Page`. On the Repo
tests (CPU) runner the top-level name resolves as a namespace directory with no
sync_api inside it, so the guard passes and the import dies with

  ImportError: cannot import name 'Page' from 'playwright.sync_api' (unknown location)

A skip condition reported as a failure, on every branch, for as long as that
runner stays that way. It is red on #9202 and #9213 too, neither of which touches
any of this.

One of the two files already said what the guard was really for: "importing a
harness pulls in playwright.sync_api". It now checks that.
danielhanchen added a commit that referenced this pull request Aug 18, 2026
…lent 30 minutes

This step stalls. Three times in one day it sat in apt's download loop until the
job's 30-minute timeout killed it, while the sibling shards finished the whole job
in 4 to 9 minutes. Twice on #9202 and once on #9189, always the same step.

The cost is out of proportion to the cause. GitHub scores a job timeout as
"cancelled" rather than a failure, prints no reason, and skips every step after
it, so the chat shard reported nothing about the chat surface for what both times
turned out to be an infrastructure hiccup that cleared on a plain re-run of the
same commit.

A per-attempt `timeout` turns the stall into a failure instead of a silent wait,
and the retry is what actually recovers. The healthy time is about 2 minutes, so 8
per attempt is 4x headroom and a merely slow mirror will not trip it.
timeout-minutes bounds the pair in case `timeout` is outlived by an unkillable
child. Same reasoning, and the same wording, as the bounded prime-hf step in
studio-mac-ui-smoke.yml.

Both install steps in this file, since ui-smoke and ui-indicator run the identical
command. Left alone on mac and windows: neither passes --with-deps, so neither has
the apt phase this is about, and neither has been observed to stall.
danielhanchen added a commit that referenced this pull request Aug 19, 2026
…t guard it (#9189)

* Repair the two chat auto-load suites that #9173 left red on main

#9173 added a vision-projector fallback to the auto-load success toast. Both
of the suites that read that code broke on it, and both were green on the
commit immediately before (54b6ca4).

72 tests, in two different shapes:

1. tests/studio/test_chat_autoload_failure_gate.py, 71 failures. The harness
   slices autoLoadSmallestModel verbatim out of chat-adapter.ts and runs it, so
   anything the slice references must exist in PREAMBLE. mmprojFallbackMessage
   did not, which is a bare ReferenceError inside the retry loop; the loop
   catches it and scores it as a failed load, so every scenario fails as a
   wrong-model assertion. The file's own guard caught this and named the symbol,
   which is what it was written for after #7699 did the same thing.

   Stubbed as a function of the reason rather than a copy of the real
   three-message record. The value only reaches `options.description`, and these
   scenarios assert on which model loaded, never on toast copy, so copying
   user-facing strings in here would give them a second home to drift from.

2. tests/studio/test_model_picker_contracts.py, 1 failure. It asserted the
   literal `description: cpuFallbackReason`, and the mmproj branch went in front
   of it. The property held; only the spelling moved. Its own comment records
   this happening once already, when the CPU-fallback branch first appeared, so
   it now pins the property: the description varies on both fallback reasons and
   still has an undefined arm for the ordinary path.

   Scoped to the description EXPRESSION, not the whole helper. `cpuFallbackReason`
   is also a parameter name in the signature above, so a substring test over the
   block stays green with the CPU branch deleted outright -- the first cut of this
   check did exactly that, and mutation caught it.

Three mutations verified red: the description no longer driven by any fallback
reason, the CPU branch dropped, and the mmproj branch dropped.

4223 passed, 4 skipped.

* Say both fallbacks when both fire, not just the projector one

Both load paths wrote the toast description as

  mmprojFallbackReason ? mmprojMessage : cpuFallbackReason ? cpuMessage : undefined

so whenever both reasons are set the CPU-fallback sentence is dropped. The user
is told "loaded without vision" and never told the model is running on the CPU,
which reads as a deliberate, explained degradation rather than an unaccelerated
session.

The combination is reachable. On a CPU-fallback replay llama_cpp.py preserves
_cpu_fallback_reason (it clears it only when not _replaying_cpu_fallback) and
resets _mmproj_fallback_reason so the projector can fail again inside that same
launch. A low VRAM machine whose Vulkan backend crashed is exactly where the
projector then falls back too.

loadFallbackNotice() in mmproj-fallback.ts is now the single composition of the
title suffix, the description and the degraded flag, and both call sites delegate
to it. CPU_FALLBACK_MESSAGE moves there as well, so the two paths cannot describe
the same condition differently again.

Tests: four combined-case cases in mmproj-fallback.test.ts, verified red against
the shipped nested ternary. test_model_picker_contracts.py now asserts the call
sites delegate and pass both reasons rather than matching the old inline ternary,
and test_chat_autoload_failure_gate.py's stub mirrors the composition so a call
site dropping a reason stays detectable.

* Repin the CPU-fallback toast test to behaviour, and unbreak the queued-capabilities test

Two frontend suites were red.

auto-load-cpu-fallback-toast.test.ts matched the warn-vs-success choice and the
message text as substrings of showAutoLoadSuccess. Both moved into
loadFallbackNotice, which is the single definition the explicit-load path now
shares, so the match went stale. Matching the inline form again would go red on a
refactor that changes nothing a user can see, and would stay green if only one of
the two load paths kept the behaviour. It now calls loadFallbackNotice and asserts
the verdict, and separately asserts the call site delegates to it.

queued-model-capabilities.test.ts was red on main before this branch. #9173 added
`import { isTextOnlyMmprojFallback } from "./mmproj-fallback"` to
image-input-support.ts, which the test imports statically. Extensionless is the
right form -- 2314 of the 2367 relative imports under src/ are written that way,
and vite and tsconfig's "bundler" mode resolve them -- but the bare node loader
does not, and a static import resolves before any registration can run. The test
now registers the bundler resolver and imports dynamically, which is what
mmproj-fallback.test.ts already does for the same module.

Both files reformatted by biome; regex literals hoisted out of the test bodies
for useTopLevelRegex.

Full frontend suite: 3734 pass, 0 fail. typecheck clean. The 4 biome errors in
chat-adapter.ts and use-chat-model-runtime.ts are byte-identical on origin/main.

* Order the split-axis abort against the mmproj strip, not against its argument

test_tensor_split_abort_raises_early_to_layer_fallback has been red on main
since #9173, which renamed the text-only strip's argument from _last_spawn_cmd
to _vision_gpu_cmd. That rename is right: the strip should read the vision GPU
command rather than whatever was spawned last, and #9173 refreshes
_last_spawn_cmd from the result immediately after. The test was pinned to the
old argument name, so a rename with no behavioural content took it down.

The failure also misreported itself. `assert raise_idx < src.find(needle)` reads
as an ordering check but is two claims at once, and when the landmark is gone it
fails with "assert 249423 < -1" -- which says the ordering broke, when what
happened is that the landmark moved. Each landmark is now required to exist
before it is ordered, and says so.

The strip is matched on the call rather than on what is passed to it. What this
test is about is that the abort raises BEFORE the projector is discarded (#6659);
which command the strip reads from is that code's own business.

Checked both ways: removing the strip call from load_model goes red with a
message naming the missing landmark, and renaming the argument again stays green.

* Name the endpoints when the heavy-thread harness catches a stray request

The harness records every /api/ URL issued during a measured action, then keeps
only the count, so the failure reads

  let 2 /api/ requests reach the network during the measured actions

and stops there. It says an interaction paid for a round trip without saying
which one, and the reader has to bisect the frontend to learn what the harness
already knew and discarded.

It now reports the endpoints. Deduplicated and capped at eight, because the case
this instrument exists to catch is a request issued once per message, which would
otherwise print hundreds of copies of one line.

This is why it surfaced now: the step has not run on main since #9173, whose unit
test break fails earlier in the same job and short-circuits it. It was last green
at 54b6ca4. With the unit tests repaired on this branch the job reaches the
step again, and the first thing it needed to say was the one thing it did not.

* Skip the playwright harness tests on the module they need, not the package

Five tests guard themselves with `pytest.importorskip("playwright")` and then
import a harness that does `from playwright.sync_api import Page`. On the Repo
tests (CPU) runner the top-level name resolves as a namespace directory with no
sync_api inside it, so the guard passes and the import dies with

  ImportError: cannot import name 'Page' from 'playwright.sync_api' (unknown location)

A skip condition reported as a failure, on every branch, for as long as that
runner stays that way. It is red on #9202 and #9213 too, neither of which touches
any of this.

One of the two files already said what the guard was really for: "importing a
harness pulls in playwright.sync_api". It now checks that.

* Stop the fork-count store asking the server about threads it has never seen

Two fixes, both found by the heavy-thread smoke once it could name what it caught.

The smoke reported "let 2 /api/ requests reach the network during the measured
actions" and, with the endpoints now printed, they were

  POST /api/chat/threads/__LOCALID_lsQbsDZ/forks

A `__LOCALID_` thread has no server record, so that request can only 404, and
getThreadForkCounts already maps 404 to the empty map the entry starts as. It is
a round trip whose answer is known before it is sent.

Not a rounding error. A new chat is in exactly that state, and this store
refreshes on CHAT_HISTORY_UPDATED_EVENT, which fires once per streaming chunk, so
the first reply in a new chat paid one useless request per debounce window for as
long as it streamed. #8992 added the store to stop the chat getting slower as a
thread fills; excluding threads the server has never seen is the same intent.
thread-ids.ts already had the predicate.

Two tests: a local thread must not fetch on subscribe or on a burst of history
events, and a saved thread on screen beside it must still refresh -- the guard has
to be per thread, not a global off switch. They import the real
isAssistantLocalThreadId rather than restating the prefix, so the rule under test
cannot drift from the app's. Both go red with the guard removed.

Second fix, same job: the playwright skip guard. The previous commit moved it from
"playwright" to "playwright.sync_api" and it still failed, because sync_api
resolves as a namespace package on that runner too. Only the symbol the harnesses
import distinguishes a usable install, so the guard now checks for Page the way
the harness does. Verified both ways against a Page-less sync_api: it skips, and
it still proceeds when Page is there.

Frontend suite 3789 pass, typecheck clean, tests/studio 4237 pass.

* Bound and retry the Playwright browser install so a stall is not a silent 30 minutes

This step stalls. Three times in one day it sat in apt's download loop until the
job's 30-minute timeout killed it, while the sibling shards finished the whole job
in 4 to 9 minutes. Twice on #9202 and once on #9189, always the same step.

The cost is out of proportion to the cause. GitHub scores a job timeout as
"cancelled" rather than a failure, prints no reason, and skips every step after
it, so the chat shard reported nothing about the chat surface for what both times
turned out to be an infrastructure hiccup that cleared on a plain re-run of the
same commit.

A per-attempt `timeout` turns the stall into a failure instead of a silent wait,
and the retry is what actually recovers. The healthy time is about 2 minutes, so 8
per attempt is 4x headroom and a merely slow mirror will not trip it.
timeout-minutes bounds the pair in case `timeout` is outlived by an unkillable
child. Same reasoning, and the same wording, as the bounded prime-hf step in
studio-mac-ui-smoke.yml.

Both install steps in this file, since ui-smoke and ui-indicator run the identical
command. Left alone on mac and windows: neither passes --with-deps, so neither has
the apt phase this is about, and neither has been observed to stall.

* Make the Playwright install retry able to actually recover

The bound added in the previous commit worked: the stall became an 8m37s step
FAILURE with a complete log instead of a silent 30-minute cancellation, and the
log named the cause on the first try.

It also showed the retry could not work. playwright shells out to apt-get as root,
so terminating the python parent leaves that child alive holding the lock, and
attempt 2 died two seconds later with

  E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 4578 (apt-get)

A retry that cannot succeed is worse than no retry: it buries the real reason
under a second, different failure. Attempt 2 now waits for the lock to clear, up
to two minutes, and only then takes it -- the holder is our own orphan and the
runner is a throwaway.

Two smaller things the same log exposed. --kill-after was missing, so a process
that ignores SIGTERM would have been waited on forever inside the step bound. And
the warning said "did not finish within 8 minutes" about a two-second exit, which
sends the next reader looking for a stall that never happened; it now separates
timeout's 124/137 from playwright refusing outright, and prints the status.

timeout-minutes 18 to 22 to cover two 8-minute attempts plus the lock wait, still
inside the job's 30.
danielhanchen and others added 3 commits August 19, 2026 04:00
broken for three main runs, and the only thing CI printed was

    urllib.error.HTTPError: HTTP Error 400: Bad Request

The response body carries llama-server's own explanation and it went out unread,
so the cause had to be reconstructed by hand from the workflow source. #9155
fixed the regression; this stops the next one costing the same dig.

The HTTPError branch of every request helper in the three inference smoke
workflows (nine sites) now prints the status, the reason and the body before
re-raising. The read itself is guarded, because a truncated or already-consumed
body must not replace the real status with a confusing one, and the original
error still propagates: reporting is not tolerating.

The tenth site, the tool-probe seed loop in studio-inference-smoke.yml, is a
caller rather than a helper. post_sse has already printed the body by the time
it re-raises there.

tests/studio/test_inference_smoke_http_diagnostics.py parses the Python actually
embedded in the workflows with ast rather than matching text, so a rewrite that
keeps the behaviour keeps passing. It also asserts every embedded probe parses at
all, which nothing else did: the heredocs are shell text inside YAML, invisible
to every linter in the repo. Seven mutations checked red (revert the change, drop
the re-raise, drop the print, drop the status code, unguard the read, remove the
handler, break the syntax).

Listed in workflow-trigger-lint because it reads workflow files, so the edit that
breaks it is workflow-only and no paths filter would collect it. Confirmed by
removing the line and watching test_workflow_guards_run_unfiltered name it.
@danielhanchen
danielhanchen merged commit d9e5c05 into main Aug 19, 2026
4 of 5 checks passed
@chatgpt-codex-connector

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant