Repository navigation
perf: run large CPU matmuls on Accelerate on Apple Silicon - #29
Conversation
The Linux x64, Linux arm64, macOS x86_64 and Windows CPU bundles were static -DGGML_NATIVE=OFF builds: AVX2/FMA on x86 (no AVX512, VNNI or AMX, and SIGILL on pre-AVX2 CPUs) and plain Armv8.0 on Linux arm64 (no dotprod or i8mm). Build them with GGML_BACKEND_DL + GGML_CPU_ALL_VARIANTS so ggml loads the best libggml-cpu-<variant> for the host at startup (Linux arm64 with GCC 14, which accepts +sme). Linux libraries get DT_RPATH $ORIGIN like the CUDA bundles, macOS ones @loader_path only. A gate checks every finished zip ships the floor variant, keeps no build-tree paths, and loads a variant from itself.
ggml's CPU backend converts Q4_0 / Q4_K / Q2_K / Q5_K / Q6_K / IQ4_NL weights to interleaved layouts (and Q4_0 / Q4_K / Q8_0 ... to AMX tiles on AMX hosts) when they are uploaded into its extra buffer types, so every matmul reading them runs a wider GEMM kernel. sd.cpp put all weights in the plain CPU buffer, so these kernels never ran. Linear tags its quantized weight as matmul-only, and the model manager places tagged weights in the first extra buffer type whose supports_op accepts the matmul. LoRA on such a weight is added to the output instead of folded into the weight; 'immediately' LoRA mode turns repacking off. SD_CPU_REPACK=0 disables it.
The auto LoRA mode looked only at the file's weight types, so an f16 file converted with --type q4_0 chose 'immediately', which turned CPU repacking off for every such run even with no LoRA. On CPU, count the conversion target as quantized and apply LoRAs at runtime, which works on repacked weights.
Measured on GitHub's Azure EPYC 9V74 (Zen 4: AVX512 without AVX-VNNI), the AVX512 variants ran sd.cpp 15 to 50% slower than the AVX2 one (MSVC and clang alike), while on Zen 5 (Strix Halo) and Emerald Rapids (AVX512 + AMX), both with AVX-VNNI, they were the fastest. Score an AVX512 variant 0 without AVX-VNNI so those hosts load the AVX2 variant, as the static build ran.
…wide repack switch
… every CPU quantize-on-load
AMX steps the shared 2D weight per activation batch (wrong batches, out-of-bounds writes) and both AMX and CPU_REPACK read strided views as contiguous rows. Make the activation contiguous and flatten every batch dim into rows before the matmul on a possibly repacked weight.
…uffer A repack buffer has no read path and holds the repacked layout, so a destination sharing its source with a repacked first destination is read from the file in its own batch.
ggml_backend_load_best opens every matching libggml-<name>-*.so before scoring it, so the current directory search ran any such library placed where sd-cli or sd-server is launched. Keep GGML_BACKEND_DIR, the executable directory and explicit search paths.
…SD libs in the rpath loops
A CPU-only workspace now schedules with ggml's BLAS backend (Accelerate on macOS) ahead of the CPU backend when the build has GGML_BLAS. The scheduler gives it the MUL_MATs it supports (large products with F32 activations and host-buffer weights); repacked weights are not host buffers and stay on the CPU repack kernels. SD_CPU_BLAS=0 turns it off. Builds without GGML_BLAS are unchanged.
The arm64 leg stays one static binary (no runtime CPU variants) and now names GGML_BLAS=ON / Apple explicitly, since sd.cpp's CPU path schedules large matmuls onto it. The Intel leg builds without BLAS so its CPU path is unchanged; the BLAS path was only measured on Apple Silicon.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a38052cd6b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (uses_blas() && node->op == GGML_OP_MUL_MAT && ggml_backend_supports_op(blas_backend_, node)) { | ||
| return true; |
There was a problem hiding this comment.
When BLAS is available and a CPU graph contains a supported MUL_MAT, this branch forces execution through the scheduler, whose path in GGMLRunner::execute_segment explicitly ignores sd_get_backend_eval_callback(). Consequently, the documented --imat-out/enable_imatrix_collection() flow on the new macOS arm64 prebuilt does not invoke collect_imatrix for these graphs and can silently save an empty or incomplete importance matrix; disable BLAS scheduling while an eval callback is installed or add callback support to scheduled execution.
AGENTS.md reference: AGENTS.md:L69-L76
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in d6736ba. When the workspace is CPU plus BLAS every tensor stays in host memory, so execute_segment now passes the eval callback to the scheduler through ggml_backend_sched_set_eval_callback; ggml's scheduler uses the same ask/collect protocol as the plain CPU path. Other scheduler setups keep the existing warning. Checked on SD1.5 at 256 px, 2 steps, with OpenBLAS: --imat-out with BLAS on used to save an empty imatrix (0 entries, the warning printed 3 times) and now saves 256 entries over 880 calls, the same tensor names and call counts as SD_CPU_BLAS=0. The rendered image is byte-identical to before.
|
@codex review |
|
@codex security review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8a408276de
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| segment_end(); | ||
| release(); | ||
| ggml_backend_free(cpu_backend_); | ||
| ggml_backend_free(blas_backend_); |
There was a problem hiding this comment.
When BLAS processes an F16 or unrepacked quantized weight, ggml_backend_blas_mul_mat retains an F32 conversion buffer as large as the biggest weight seen. Because this backend is freed only by the workspace destructor, GGMLRunner::runner_end() releases the scheduler and model weights but leaves that scratch resident; running the conditioner, diffusion model, and VAE sequentially can therefore accumulate one large, untracked buffer per live runner and exhaust RAM on memory-constrained Macs. Reclaim or recreate the BLAS backend at runner-session boundaries rather than retaining it until model destruction.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 0fab806. ggml-blas only frees work_data when the backend is freed, so runner_end() now frees and recreates the idle BLAS backend right after releasing the workspace (thread count is set again on every segment). Within a run the buffer is kept: the diffusion runner does not end between sampling steps, so a txt2img run releases it three times (after the conditioner, after sampling, after the VAE). Peak RSS on SD1.5 at 256 px, 2 steps, CPU with OpenBLAS, 5 runs each: 2525 to 2564 MB before, 2475 to 2500 MB after. Rendered images are byte-identical with BLAS on and with SD_CPU_BLAS=0.
With BLAS scheduling a CPU graph goes through ggml_backend_sched, which ignored sd_get_backend_eval_callback(), so --imat-out collected nothing. Pass the callback to the scheduler when the workspace is CPU + BLAS only.
ggml-blas keeps its F32 conversion buffer, sized to the largest non-F32 weight it has seen, until the backend is freed. Recreate the BLAS backend in runner_end() so a finished runner does not keep it resident.
|
@codex review |
1 similar comment
|
@codex review |
# Conflicts: # scripts/unsloth/cpu_variants_gate.sh # scripts/unsloth/ggml-patches/0009-ggml-backend-no-cwd-search.patch # src/core/ggml_extend.cpp # src/core/ggml_extend_backend.cpp # src/core/ggml_extend_backend.h # src/model_manager.h # src/pipeline/diffusion_engine.cpp
1 participant
Stacked on #28 (base
perf/cpu-isa): the speedups below include #28's weight repacking.Problem
When Studio runs a GGUF image model through sd.cpp on a Mac on the CPU backend (the fallback after Metal aborts on an op it cannot run, or
--backend cpu), it uses the arm64 prebuilt's CPU path. #28 adds runtime CPU variants and weight repacking, but on Apple Silicon theapple_m1variant ran about 40% slower than the current static binary, so the macOS arm64 leg was left alone. Macs therefore got almost nothing from #28.There were two reasons:
apple_m1variant is built withDOTPRODand noFP16_VECTOR_ARITHMETIC. The static build already getsNEON | ARM_FMA | FP16_VA | DOTPRODfrom Apple clang's default target, so the variant was a downgrade. No-mcpuchange is needed.GGML_BLASis on by default on Apple), but sd.cpp only ever scheduled the CPU backend, so Accelerate was never used for matmuls.Change
src/core/compute_workspace.*,src/core/ggml_extend_backend.*,src/core/ggml_runner.cpp: a CPU-only workspace now schedules with ggml's BLAS backend ahead of the CPU backend whenever the build includes it. The scheduler gives BLAS the MUL_MATs it supports: large products with F32 activations whose weights sit in host buffers. Weights that CPU bundles: runtime-selected ggml CPU variants and repacked quantized weights (1.3x to 3x faster on CPU) #28 repacked are not in host buffers, so their matmuls stay on the repack kernels.SD_CPU_BLAS=0turns this off. Builds withoutGGML_BLAS(Linux, Windows) are unchanged. The first few graphs logcpu blas: N of M graph nodes run on BLASso you can see when it engages.unsloth-sd-prebuilt.yml: the macOS arm64 leg stays a single static binary with no CPU variants and now sets-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Appleexplicitly. The macOS Intel leg gets-DGGML_BLAS=OFF, which keeps its CPU path as it is today, because the BLAS path was only measured on Apple Silicon.Results
Hardware: GitHub
macos-15runners (Apple M1, virtual, 3 cores). Backend: CPU (--backend cpu). Base is fork masterafac7a7c, built exactly like the arm64 release leg (static, Metal embedded,GGML_NATIVE=OFF). Head is this branch built the same way. Each measurement is a fresh process, arms run in ABBA order, threads are left at sd.cpp's default (3). Ratios are base/head computed per repeat on the same runner; the table shows the median with [min, max].SD-Turbo, 512x512, 2 steps. 9 paired repeats across 3 runners:
Noise floor: on f16 with
SD_CPU_BLAS=0, head produces an image bit-identical to base. That arm's timing ratio was 1.13 [0.67, 1.46], so the f16 timing is a tie. The quantized types have minimums above 1.1 on every repeat.Contribution of each part to sampling speedup (median):
Z-Image-Turbo Q4_K_M, with Qwen3-4B Q4_K_M as the text encoder and the
aeVAE, 256x256, 2 steps. 3 paired repeats:The Z-Image head image is 35.7 dB PSNR against base.
The VAE decode at 512 (about 30 s of the roughly 55 s SD-Turbo total) is unchanged, so the end-to-end gain on SD-Turbo is smaller than the sampling gain.
Tests
--type f32, same prompt and seed. f16 and q4_K move closer to it. q4_0 moves -0.1 dB. q8_0 moves -1.2 dB, but the BLAS-only q8_0 arm, which does strictly more precise math (f32 activations), also scores lower than base (39.57 vs 40.11). At this level, single-image PSNR is reacting to small perturbations, not to precision.REPACKtensors andcpu blas: N of M graph nodes. WithSD_CPU_BLAS=0the count is 0 and the f16 output matches base bit for bit.Not covered
--backend cpu. That took about 35 s on the first run and was cached afterwards.macos-14runner, which this PR does not change.