Unblock Mac Studio GGUF CI, red on every main run since #8883 - #9155
Conversation
Mac Studio GGUF CI has failed on every main run from ee68d9e onward, the merge of #8883 (refuse a gguf that cannot fit in free vram plus available ram). Bisected over 90 completed main runs: green through 31c42e8, red from ee68d9e, no recovery since. The guard is not wrong. From the failing run's own server log: Metal device looks virtualised (Apple Paravirtual device) ... GGUF inference will run on CPU Forcing gpu_layers=0 for gemma-4-E2B-it-UD-Q4_K_XL.gguf Error loading model: About 3 GB of this model does not fit in GPU memory and would run from system RAM. Only about 4 GB is available and 2 GB of that is kept free for the rest of the system, leaving about 2 GB usable. GitHub's macOS runners expose a paravirtual Metal device, and Unsloth already refuses to offload to one because paravirtual Apple GPUs return corrupt output. So the launch is --gpu-layers 0 --device none and the WHOLE model is a host mapping, not the partial spill the guard was written to price. It then measures honestly: gemma-4-E2B UD-Q4_K_XL (~2.4 GB) plus mmproj-F16 (~780 MB) does not fit in 2 GB usable. The load returns HTTP 400 and the phase fails several layers away from the cause. The workflow is what was wrong. It had been getting away with a model the runner cannot hold because the prompts are tiny and the mapping is paged, which is exactly the gamble the guard stops taking on a user's machine. CI now takes it knowingly with UNSLOTH_ALLOW_HOST_OFFLOAD, the escape hatch the refusal names, rather than silently swapping in a smaller model and quietly changing what mac vision inference is tested against. Set at job level, not on the vision phase: every phase there is CPU-only for the same reason, so a runner image with slightly less free RAM would take out the tool-calling phase next. tests/studio/test_mac_host_offload_optin.py keeps the blast radius honest. The opt-out disables a real safety net, so it asserts the reason stays written down next to it, that the Linux and Windows GGUF workflows never acquire it -- there a silenced guard would let a genuine host-offload regression pass green -- and that the guard keeps its own unit tests.
for more information, see https://pre-commit.ci
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a567b93472
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| return yaml.safe_load((WORKFLOWS / name).read_text(encoding = "utf-8")) | ||
|
|
||
|
|
||
| def test_the_mac_gguf_job_opts_out_at_job_level(): |
There was a problem hiding this comment.
This test is auto-collected by Backend CI, but that workflow's pull_request.paths excludes .github/workflows/**; Core has the same omission, while the unfiltered workflow-trigger-lint.yml invokes several workflow invariant tests explicitly but not this file. Consequently, a PR that removes this opt-in or adds it to the Linux/Windows workflow—the exact changes these assertions are intended to reject—does not run this guard before merge. Wire this test into an unfiltered workflow such as Workflow trigger lint.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed: studio-backend-ci.yml's pull_request.paths cover studio/, unsloth/, unsloth_cli/, tests/, install.sh, install.ps1, scripts/**, pyproject.toml and its own YAML, so a PR that only edits studio-mac-inference-smoke.yml never starts that job, and workflow-trigger-lint carries no paths filter at all. Fixed in 22d5514 by wiring it in there, next to the other workflow invariants that live in that job for exactly this reason.
| def test_no_other_gguf_workflow_disables_the_guard(name): | ||
| doc = _doc(name) | ||
| offenders = [] | ||
| for jid, job in (doc.get("jobs") or {}).items(): |
There was a problem hiding this comment.
When either non-Mac workflow sets UNSLOTH_ALLOW_HOST_OFFLOAD in its top-level env, GitHub propagates it to every job, but this loop inspects only job and step environments and therefore reports no offender. That leaves a straightforward way to disable the safety guard across the entire Linux or Windows workflow while this blast-radius test remains green; inspect doc.get("env") as well.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct, and it was the widest hole in a test whose subject is blast radius: a top-level env propagates to every job in the workflow. Fixed in 22d5514 by checking doc's own env before the job and step scan. Verified by adding UNSLOTH_ALLOW_HOST_OFFLOAD at the top of studio-inference-smoke.yml, which now fails the guard where it previously passed.
…evel env The guard rejects a PR that removes UNSLOTH_ALLOW_HOST_OFFLOAD from the macOS GGUF job or adds it to the Linux or Windows one. Such a PR edits a workflow and nothing else, which Backend CI's paths filter does not match, so the assertion was never collected for the one change it exists to reject. Wired into workflow-trigger-lint, which carries no paths filter and already hosts the other workflow invariants for this reason. The blast-radius check also read only job and step env. GitHub propagates a top-level env to every job, so one line at the top of the Linux or Windows workflow would have disabled the guard across all of them while this test stayed green.
|
@codex review |
broken for three main runs, and the only thing CI printed was
urllib.error.HTTPError: HTTP Error 400: Bad Request
The response body carries llama-server's own explanation and it went out unread,
so the cause had to be reconstructed by hand from the workflow source. #9155
fixed the regression; this stops the next one costing the same dig.
The HTTPError branch of every request helper in the three inference smoke
workflows (nine sites) now prints the status, the reason and the body before
re-raising. The read itself is guarded, because a truncated or already-consumed
body must not replace the real status with a confusing one, and the original
error still propagates: reporting is not tolerating.
The tenth site, the tool-probe seed loop in studio-inference-smoke.yml, is a
caller rather than a helper. post_sse has already printed the body by the time
it re-raises there.
tests/studio/test_inference_smoke_http_diagnostics.py parses the Python actually
embedded in the workflows with ast rather than matching text, so a rewrite that
keeps the behaviour keeps passing. It also asserts every embedded probe parses at
all, which nothing else did: the heredocs are shell text inside YAML, invisible
to every linter in the repo. Seven mutations checked red (revert the change, drop
the re-raise, drop the print, drop the status code, unguard the read, remove the
handler, break the syntax).
Listed in workflow-trigger-lint because it reads workflow files, so the edit that
breaks it is workflow-only and no paths filter would collect it. Confirmed by
removing the line and watching test_workflow_guards_run_unfiltered name it.
* Print what the server said when an inference smoke request 4xx's
broken for three main runs, and the only thing CI printed was
urllib.error.HTTPError: HTTP Error 400: Bad Request
The response body carries llama-server's own explanation and it went out unread,
so the cause had to be reconstructed by hand from the workflow source. #9155
fixed the regression; this stops the next one costing the same dig.
The HTTPError branch of every request helper in the three inference smoke
workflows (nine sites) now prints the status, the reason and the body before
re-raising. The read itself is guarded, because a truncated or already-consumed
body must not replace the real status with a confusing one, and the original
error still propagates: reporting is not tolerating.
The tenth site, the tool-probe seed loop in studio-inference-smoke.yml, is a
caller rather than a helper. post_sse has already printed the body by the time
it re-raises there.
tests/studio/test_inference_smoke_http_diagnostics.py parses the Python actually
embedded in the workflows with ast rather than matching text, so a rewrite that
keeps the behaviour keeps passing. It also asserts every embedded probe parses at
all, which nothing else did: the heredocs are shell text inside YAML, invisible
to every linter in the repo. Seven mutations checked red (revert the change, drop
the re-raise, drop the print, drop the status code, unguard the read, remove the
handler, break the syntax).
Listed in workflow-trigger-lint because it reads workflow files, so the edit that
breaks it is workflow-only and no paths filter would collect it. Confirmed by
removing the line and watching test_workflow_guards_run_unfiltered name it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Collapse the split f-string the formatter left behind
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 participant
Mac Studio GGUF CI has failed on every main run from ee68d9e onward, the merge
of #8883 (refuse a gguf that cannot fit in free vram plus available ram). Bisected
over 90 completed main runs: green through 31c42e8, red from ee68d9e, no
recovery since.
The guard is not wrong. From the failing run's own server log:
Metal device looks virtualised (Apple Paravirtual device) ... GGUF inference
will run on CPU
Forcing gpu_layers=0 for gemma-4-E2B-it-UD-Q4_K_XL.gguf
Error loading model: About 3 GB of this model does not fit in GPU memory and
would run from system RAM. Only about 4 GB is available and 2 GB of that is
kept free for the rest of the system, leaving about 2 GB usable.
GitHub's macOS runners expose a paravirtual Metal device, and Unsloth already
refuses to offload to one because paravirtual Apple GPUs return corrupt output.
So the launch is --gpu-layers 0 --device none and the WHOLE model is a host
mapping, not the partial spill the guard was written to price. It then measures
honestly: gemma-4-E2B UD-Q4_K_XL (~2.4 GB) plus mmproj-F16 (~780 MB) does not fit
in 2 GB usable. The load returns HTTP 400 and the phase fails several layers away
from the cause.
The workflow is what was wrong. It had been getting away with a model the runner
cannot hold because the prompts are tiny and the mapping is paged, which is
exactly the gamble the guard stops taking on a user's machine. CI now takes it
knowingly with UNSLOTH_ALLOW_HOST_OFFLOAD, the escape hatch the refusal names,
rather than silently swapping in a smaller model and quietly changing what mac
vision inference is tested against.
Set at job level, not on the vision phase: every phase there is CPU-only for the
same reason, so a runner image with slightly less free RAM would take out the
tool-calling phase next.
tests/studio/test_mac_host_offload_optin.py keeps the blast radius honest. The
opt-out disables a real safety net, so it asserts the reason stays written down
next to it, that the Linux and Windows GGUF workflows never acquire it -- there a
silenced guard would let a genuine host-offload regression pass green -- and that
the guard keeps its own unit tests.