Repository navigation
Studio: pin sd.cpp to the mirror build with Windows CUDA, sharing llama.cpp's CUDA runtime - #13175
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@codex review |
1 similar comment
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3eb91808e3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| resident = ensure_sd_server_binary( | ||
| allow_install = False, accelerator = accelerator | ||
| ) or ensure_sd_cpp_binary(allow_install = False, accelerator = accelerator) | ||
| return bool(resident) and _pin_moved(resident, accelerator) |
There was a problem hiding this comment.
When the old managed build is currently running (for example, the user switches from another native family to Qwen-Image-2.1), _pin_moved() returns false because _managed_tree_in_use() is true. The router therefore predicts and selects diffusers, and only afterward _activate() unloads the native model; unlike accelerator changes, no pin-upgrade retry occurs after teardown. Thus the first load still stages the dense diffusers assets and does not install the new fallback, defeating this fix until a subsequent load. The stale-pin check needs a deferred-after-teardown path rather than treating an in-use tree as a build that will not upgrade.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Not reachable in practice: the stale build is replaced by the first native load in a Studio process, whatever its family, because the managed tree is free at that point (_pin_moved only stands down while something runs out of the tree). So any native model resident afterwards is already on the new build. The in-use plus stale combination only exists after that upgrade failed, where the once-per-process memo deliberately stops retrying.
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
danielhanchen
changed the title
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bce5bd00e3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| runtime = any(n.startswith(("cudart64_", "libcudart.so")) for n in names) | ||
| return cuda and not runtime |
There was a problem hiding this comment.
When an existing CUDA 12 managed install is upgraded to the new CUDA 13 bundle, extraction merges into the old tree and _discard_superseded_binaries() removes only sd-cli/sd-server, leaving the old libcudart.so.12 or cudart64_12.dll beside the replacement. This check treats any cudart major as a complete bundled runtime, so runtime_env() does not add llama.cpp's CUDA 13 directories and the new ggml library cannot resolve its CUDA 13 cublas/runtime dependencies. This particularly breaks the moved-pin upgrade this commit introduces; determine whether the matching runtime is present or remove superseded runtime libraries during replacement.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Not reproducible: every release zip extracts under its own sd--bin-/ directory, so an upgraded install's bindir holds only the new bundle. Installed the real old pin (u1d02858 cuda12, with libcudart.so.12) through Studio, then upgraded with this head to the cuda13 bundle: the old libcudart.so.12 stays in sd-master-813-...-cuda12/, the new bindir has no cudart, _needs_host_cuda_runtime is True, runtime_env puts nvidia/cu13/lib on the path and sd-cli --list-devices finds the GPU.
| binary = _llama_server_binary() | ||
| if not binary: | ||
| return [] |
There was a problem hiding this comment.
On the supported off-torch route (for example, UNSLOTH_DIFFUSION_SD_CPP_DEVICE=nvidia:0 beside a ROCm PyTorch/llama.cpp install), there may be no CUDA llama-server runtime to borrow and merely having an NVIDIA driver does not provide cudart/cublas. The new mirror CUDA archives deliberately omit those libraries, but this implementation derives search paths only from the installed llama-server environment, so the freshly installed sd.cpp binary fails its loader probe on such hosts whereas the previous self-contained CUDA bundle worked. The CUDA install needs an independent runtime source or must select a self-contained fallback when llama.cpp is absent or non-CUDA.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Not addressed here by design: Studio always installs unslothai/llama.cpp, and on an NVIDIA host its CUDA build is what supplies cudart / cublas; the CUDA bundles intentionally reuse that runtime instead of shipping another copy. A host forcing sd.cpp onto an NVIDIA card while torch and llama.cpp are ROCm builds is outside that setup.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 67358f2ffc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if accel == "cuda": | ||
| sel = _prefer_cuda_major(sel, cuda_major or "12") | ||
| if sel: |
There was a problem hiding this comment.
When a Windows release contains both CUDA 12/13 engine archives plus the standard cudart-sd-...-cu12...zip, selecting the CUDA 13 archive here is followed by _maybe_fetch_windows_cudart() unconditionally extracting the first Windows cudart helper, regardless of chosen. The resulting CUDA 13 install gains CUDA 12 DLLs; _needs_host_cuda_runtime() then sees cudart64_12.dll and declines to add llama.cpp's CUDA 13 runtime paths, so the CUDA 13 binary cannot resolve its cublas 13 dependencies. Filter the helper by the CUDA major in the selected archive, or skip it for the new runtime-less mirror bundles.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
No published release has that shape. The only release with a cuda13 Windows archive is the pinned mirror release, and it ships no cudart helper. Every leejet release (the fallback) publishes only win-cuda12 next to the cu12 cudart archive. The mirror archives also extract into their own sd--bin-/ directory while the cudart helper extracts into the install root, so _needs_host_cuda_runtime(bindir) would not see a cudart64_12.dll even if one were fetched.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 67358f2ffc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| major = host_cuda_runtime_major() | ||
| if major: | ||
| kwargs["cuda_major"] = major |
There was a problem hiding this comment.
When the managed llama.cpp installation switches CUDA lines (for example, CUDA 12 to CUDA 13) without this sd.cpp pin changing, the existing runtime-less sd.cpp bundle is reused because _needs_reinstall() records and compares only the cuda accelerator class and pin. runtime_env() then supplies the new llama.cpp CUDA 13 directories to an sd.cpp binary still linked against CUDA 12 sonames, so native loading fails until the user deletes the managed sd.cpp tree. Persist the selected CUDA major/asset in the install record and include it in the reinstall decision.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Already handled by the existing unrunnable-binary repair. sd-cli links libggml.so.0, which links libggml-cuda.so.0, so a bundle whose CUDA major is no longer on the search path fails at exec: the real master-951 cuda13 sd-cli exits 127 with "libcublasLt.so.13: cannot open shared object file" (Windows reports 0xC0000135, also in _WINDOWS_IMAGE_LOAD_FAILURE_STATUSES). _server_binary_runnable probes with runtime_env, returns False, and _usable_or_discard_managed removes the managed copy, so ensure_sd_cpp_binary reinstalls with _install_kwargs. Reproduced with the real cuda13 bundle in a managed tree whose record names the current pin, with no CUDA 13 dirs and llama.cpp on cuda12: the next ensure_sd_cpp_binary(accelerator="cuda") called install(accelerator="cuda", cuda_major="12"). If the old major's libraries are still present, the bundle keeps running on them.
|
@codex review |
Fixes the native engine bug in #12470 (the engine / text encoder picker that issue also asks for is a separate feature and is not part of this PR).
The bug
The pinned sd.cpp release (
master-813-bfbef5b-u1d02858) on the Unsloth mirror has no Windows CUDA, Windows ROCm or Linux ROCm asset. On those hosts the installer falls back to upstream at the pin's base,master-813-bfbef5b(2026-08-05). Qwen-Image-2.1 landed upstream inmaster-883(137f740, 2026-09-20), so that build cannot run it.What the reporter saw on Windows + CUDA with the native engine:
leejet/stable-diffusion.cpp master-813-bfbef5b, the router's architecture gate finds noqwen_image_2_1in it, logspredates qwen-image-2.1 support; using diffusers, and the load pulls the ~17 GB dense text encoder instead.The fix
DEFAULT_TAGmoves tomaster-951-f89d9b1-uafac7a7, the mirror release from Publish Windows CUDA bundles and leave the CUDA runtime to the host stable-diffusion.cpp#25. It is built on upstreammaster-951, which carries Qwen-Image-2.1 and its reference-alpha fix (master-896), and it now ships Windows CUDA (win-cuda12-x64,win-cuda13-x64) and Linux CUDA 12 / 13 bundles. Hosts the mirror still does not build (ROCm) fall back to upstreammaster-951-f89d9b1, whose Linux and Windows ROCm builds carry every family marker (qwen_image_2_1,qwen_image_layers, the 2.1 edit marker); Studio's own family check accepts the Linux ROCm install for Qwen-Image-2.1.u1d02858, or leejetmaster-813-bfbef5bon the fallback hosts) is replaced once on the next load by the existing pin-upgrade path (kill switchUNSLOTH_SD_CPP_AUTO_UPGRADE=0, once-per-process failure memo).predict_engine): when the resident managed build is about to be replaced for a moved pin, the plan predicts what the load will do (native) instead of planning diffusers off the build that is about to be removed. Without this the upgraded host would stage the dense text encoder once and then load natively.CUDA bundles share llama.cpp's CUDA runtime
The mirror's CUDA bundles no longer carry cudart / cublas / cublasLt, and ggml is a shared library, so sd-cli and sd-server share one copy of the CUDA kernels. Studio always installs unslothai/llama.cpp, whose CUDA build already needs and resolves the same runtime.
install_sd_cpp_prebuilt.py: a CUDA install prefers thecuda<major>asset matching the host runtime (--cuda-majoron the CLI). A release without that major still offers its CUDA build, so older self-contained releases and upstream are unchanged; Windows with an unknown runtime keeps cuda12 as before.sd_cpp_backend._install_kwargs: the major is theruntime_lineof the installed llama.cpp build, else torch's CUDA version. Non-CUDA installs get exactly the same arguments as before.sd_cpp_engine.runtime_env: a CUDA bundle that shipsggml-cudabut nocudartgets the library directoriesllama-servergets (LlamaCppBackend._llama_server_env_for_binary), minus llama.cpp's own build dir off Windows since itslibggml*.soshare sonames with the bundle's (the bundle's DT_RPATH$ORIGINwins regardless). Every other bundle gets the same environment as before.Reusing llama.cpp's
ggml-cudaitself is not possible: the mirror's ggml carries patches (fused H3 ops, cuDNN attention, ...) that llama.cpp's build does not have, so only the CUDA runtime is shared.ggml-cudalinks from the hostLinux-Ubuntu-22.04-x86_64-cuda12Linux-Ubuntu-22.04-x86_64-cuda12libcudart/libcublas/libcublasLt.so.12Linux-Ubuntu-22.04-x86_64-cuda13libcudart/libcublas/libcublasLt.so.13win-cuda12-x64cudart64_12,cublas64_12,cublasLt64_12(old pin: upstream's 322 MiB build plus a 537 MiB cudart archive)win-cuda13-x64cublas64_13,cublasLt64_13(cudart static, as in unslothai/llama.cpp's cuda13 build)Every Windows DLL / EXE in the new bundles carries an Authenticode signature.
Verification
Reproduced on main (Linux x86_64,
--accelerator vulkanandrocm, the same upstream fallback Windows CUDA took):Reporter state (a managed tree holding
leejet master-813-bfbef5bunder the old pin),UNSLOTH_DIFFUSION_ENGINE=sd_cpp,unsloth/Qwen-Image-2.1-GGUFQ4_K_M:the installed sd.cpp build predates 'qwen-image-2.1' supportQwen-Image-2.1 Q4_K_M through Studio's native engine with the new release's Linux bundles (built from the same tree), 512x512, 12 steps, seed 3407, on a B200:
selected: sd_cpp, no fallback. cuda13 loaded in 95 s and generated in 36 s, withlibcudart.so.13/libcublas.so.13mapped from the venv'snvidia/cu13/lib; cuda12 loaded in 70 s and generated in 26 s against the host's CUDA 12.8 runtime. The two images are pixel-identical, peak 11.4 GB.Live resolution against GitHub, previous head of this PR vs this head (30 host x accelerator cells, 22 move): every mirror host moves
u1d02858->uafac7a7; Windows CUDA, Windows Vulkan and Linux Vulkan move from the upstream fallback to the mirror (win-cuda12-x64,win-vulkan-x64,Linux-Ubuntu-22.04-x86_64-vulkan); ROCm moves to upstreammaster-951-f89d9b1.master-951-f89d9b1ROCm (the fallback for hosts the mirror skips), installed through Studio's installer:sd_cpp_binary_runs_family(..., qwen-image-2.1)is True for both sd-cli and sd-server.sd-cli --help, old pin vs new: 168 flags each;--cannyand--disable-auto-resize-ref-imageare gone (Studio passes neither),--conditioning-cache-sizeand--image-preprocessare new.libcublasLt.so.13: cannot open shared object file; with Studio'sruntime_envthey loadnvidia/cu13/lib/{libcudart,libcublas,libcublasLt}.so.13(the files llama-server uses) and find the GPU. With llama.cpp's build dir first on the path they still load their ownlibggml*.so.ensure_sd_server_binary(accelerator="cuda")against the new assets picks the bundle matching llama.cpp'sruntime_line, extracts sd-cli, sd-server and thelibggml*.so.0, and the result runs--list-devices(CUDA0 NVIDIA B200).Old pin (
u1d02858) vs new pin (uafac7a7), Linux cuda12 bundles through Studio's native engine, same tree, seed 3407, 512x512, B200 (selected: sd_cppon every run, peak 13 GB):All six images are coherent renders of the prompt. Z-Image moves the most (same subject and scene, slightly different pose and framing); the only change to its default path in this range is upstream's f16 overflow fix for Z-Image quantized matmuls on CUDA (leejet/stable-diffusion.cpp#2074), which rescales the quantized output projection.
Upgrade in place, live: the old pin's self-contained cuda12 bundle installed through Studio, then this head's
ensure_sd_server_binary(accelerator="cuda")with a cuda13 host runtime: the new bundle lands in its ownsd-<tag>-bin-<label>/directory (the oldlibcudart.so.12stays in the old one),runtime_envaddsnvidia/cu13/lib, andsd-cli --list-devicesfinds the B200.Switchboard (base 4527f33 vs head bce5bd0, 11 targets: auth, settings, upstream_ci, inference_settings, diffusion_gen, diffusion_video, crawl, diffusion bench tiny http / inproc / matrix, diffusion imports): NO_REGRESSION, 67 SAME, 1 VISUAL_DIFF on the About tab, which is only the llama.cpp version string (
b11505vsb11541, the head install pulled a newer llama.cpp prebuilt).Not run: no Windows GPU was available, so the Windows CUDA bundles are verified by contents, imports and signatures only.