{{ message }}
Repository navigation
Tags: unslothai/stable-diffusion.cpp
Tags
feat(server): add recipe-based model management (leejet#2176)
Layer split: size device ranges in proportion to capacity and keep co… …mpute headroom under --max-vram (#32) The partitioner filled the first device up to its weight capacity before using the next one, so a module whose weights fit the main device alone was never split, and the compute buffers then had no room under the budget. The 2 GiB compute headroom was also taken only from free memory, never from an explicit --max-vram budget, so a per-device budget was filled entirely with weights. Ranges are now sized in proportion to each device's capacity (the lower of free memory and its --max-vram budget, minus the headroom), as docs/backend.md already described, with the fill-first plan kept as a fallback when the proportional ranges do not fit. The planner is a header-only function with a standalone test (scripts/layer_split_ranges_test.cpp).
feat: add FLUX.3 Image Klein and Klein Base support (leejet#2143)
perf: run large CPU matmuls on Accelerate on Apple Silicon (#29) * Ship every ggml CPU variant in the CPU bundles The Linux x64, Linux arm64, macOS x86_64 and Windows CPU bundles were static -DGGML_NATIVE=OFF builds: AVX2/FMA on x86 (no AVX512, VNNI or AMX, and SIGILL on pre-AVX2 CPUs) and plain Armv8.0 on Linux arm64 (no dotprod or i8mm). Build them with GGML_BACKEND_DL + GGML_CPU_ALL_VARIANTS so ggml loads the best libggml-cpu-<variant> for the host at startup (Linux arm64 with GCC 14, which accepts +sme). Linux libraries get DT_RPATH $ORIGIN like the CUDA bundles, macOS ones @loader_path only. A gate checks every finished zip ships the floor variant, keeps no build-tree paths, and loads a variant from itself. * Repack quantized matmul weights for the CPU backend ggml's CPU backend converts Q4_0 / Q4_K / Q2_K / Q5_K / Q6_K / IQ4_NL weights to interleaved layouts (and Q4_0 / Q4_K / Q8_0 ... to AMX tiles on AMX hosts) when they are uploaded into its extra buffer types, so every matmul reading them runs a wider GEMM kernel. sd.cpp put all weights in the plain CPU buffer, so these kernels never ran. Linear tags its quantized weight as matmul-only, and the model manager places tagged weights in the first extra buffer type whose supports_op accepts the matmul. LoRA on such a weight is added to the output instead of folded into the weight; 'immediately' LoRA mode turns repacking off. SD_CPU_REPACK=0 disables it. * Pick at_runtime LoRA on CPU when the weights end up quantized The auto LoRA mode looked only at the file's weight types, so an f16 file converted with --type q4_0 chose 'immediately', which turned CPU repacking off for every such run even with no LoRA. On CPU, count the conversion target as quantized and apply LoRAs at runtime, which works on repacked weights. * Offer ggml's AVX512 CPU variants only on CPUs with AVX-VNNI Measured on GitHub's Azure EPYC 9V74 (Zen 4: AVX512 without AVX-VNNI), the AVX512 variants ran sd.cpp 15 to 50% slower than the AVX2 one (MSVC and clang alike), while on Zen 5 (Strix Halo) and Emerald Rapids (AVX512 + AMX), both with AVX-VNNI, they were the fastest. Score an AVX512 variant 0 without AVX-VNNI so those hosts load the AVX2 variant, as the static build ran. * Number the AVX512 gate patch 0008 so it applies after the 0007 GPU patch * CPU repack: judge an allocated weight by its buffer, not the process-wide repack switch * CPU repack: keep the LoRA-merge opt-out per context; runtime LoRA for every CPU quantize-on-load * LoRA auto mode: count TE and VAE routed to the CPU, not only the DiT * CPU repack: say when --mmap keeps quantized weights out of the repacked buffers * LoRA auto mode: only tensor-type rules that quantize force runtime LoRA * CPU repack: give repacked matmuls contiguous 2D activation rows AMX steps the shared 2D weight per activation batch (wrong batches, out-of-bounds writes) and both AMX and CPU_REPACK read strided views as contiguous rows. Make the activation contiguous and flatten every batch dim into rows before the matmul on a possibly repacked weight. * Model loading: never copy a duplicate destination from a CPU repack buffer A repack buffer has no read path and holds the repacked layout, so a destination sharing its source with a repacked first destination is read from the file in its own batch. * ggml: do not search the current directory for dynamic backends ggml_backend_load_best opens every matching libggml-<name>-*.so before scoring it, so the current directory search ran any such library placed where sd-cli or sd-server is launched. Keep GGML_BACKEND_DIR, the executable directory and explicit search paths. * cpu_variants_gate: clean up temp dirs on early failure; cover shared SD libs in the rpath loops * Trim comments * CPU: hand large matmuls to ggml's BLAS backend when it is built in A CPU-only workspace now schedules with ggml's BLAS backend (Accelerate on macOS) ahead of the CPU backend when the build has GGML_BLAS. The scheduler gives it the MUL_MATs it supports (large products with F32 activations and host-buffer weights); repacked weights are not host buffers and stay on the CPU repack kernels. SD_CPU_BLAS=0 turns it off. Builds without GGML_BLAS are unchanged. * macOS prebuilts: Accelerate BLAS on the arm64 leg, off on the Intel leg The arm64 leg stays one static binary (no runtime CPU variants) and now names GGML_BLAS=ON / Apple explicitly, since sd.cpp's CPU path schedules large matmuls onto it. The Intel leg builds without BLAS so its CPU path is unchanged; the BLAS path was only measured on Apple Silicon. * Trim comments for CPU BLAS scheduling * CPU BLAS: run eval callbacks on the scheduled CPU path With BLAS scheduling a CPU graph goes through ggml_backend_sched, which ignored sd_get_backend_eval_callback(), so --imat-out collected nothing. Pass the callback to the scheduler when the workspace is CPU + BLAS only. * CPU BLAS: release the BLAS conversion scratch at runner end ggml-blas keeps its F32 conversion buffer, sized to the largest non-F32 weight it has seen, until the backend is freed. Recreate the BLAS backend in runner_end() so a finished runner does not keep it resident. * Trim comments
feat: add Iris-3B text-to-image support (leejet#2117)
Ship current Vulkan prebuilts and restore Volta CUDA support (#22) * fix: include Volta kernels in Linux CUDA prebuilts * fix: ship current native Vulkan builds for Linux and Windows * ci: verify Vulkan SDKs and target Ubuntu 22.04
PreviousNext