iframe-proxy

akleine · 2026-10-18T10:21:13Z

If runnnig LoRAs using --lora-apply-mode auto the automatic decision between modes "immediately" and "at_runtime" works very well, but there is still a minor gap if using command line options like --type q4_0 etc.

This PR should solve the problem by checking the --type option.

Summary

Two examples will show the problem:

This first runs very well in at_runtime mode:

./sd-cli  -m ~/SD_models/sd1/picxReal_10.safetensors_Q4_0.gguf  \
  --lora-model-dir ~/SD_models/loras_sd15 \
  -pL "A cute cat<lora:LCM-lora-sdv1-5:1>" \
  --sampling-method lcm --steps 4 --cfg-scale

The sceond is similar but will crash because the --lora-apply-mode is is wrong state.
It runs in immediately mode - but should not:

./sd-cli  -m ~/SD_models/sd1/picxReal_10.safetensors \
  --lora-model-dir ~/SD_models/loras_sd15 \
  -p "A cute cat<lora:LCM-lora-sdv1-5:1>" \
  --sampling-method lcm --steps 4 --cfg-scale 1 \
  --type q4_0

At all: using --type q4_0 is the difference.

Related Issue / Discussion

Additional Information

#3  0x00005cc43b03dbd7 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) ()
#4  0x00005cc43b051580 in ggml_cuda_mul_mat_cublas(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggmusing command line options like  --type q4_0 l_tensor*) ()
#5  0x00005cc43b0567f2 in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) ()
#6  0x00005cc43be91ce5 in ggml_backend_graph_compute ()
#7  0x00005cc43aad6bc8 in GGMLRunner::execute_segment(ggml_cgraph*, int) ()
#8  0x00005cc43aada37a in GGMLRunner::execute_graph(ggml_cgraph*, int, bool, std::function<bool ()> const&) ()
#9  0x00005cc43aadbd2f in GGMLRunner::compute(std::function<ggml_cgraph* ()>, int, bool, bool, std::function<bool ()> const&) ()
#10 0x00005cc43aa6fd24 in LoraModel::apply(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ggml_tensor*, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ggml_tensor*> > >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, SDVersion, int, bool) ()
#11 0x00005cc43aa6cda8 in ModelManager::apply_loras_to_params(std::vector<ModelManager::TensorState*, std::allocator<ModelManager::TensorState*> > const&) ()

Checklist

I have read and confirmed this PR follows the contribution guidelines.

Summary by CodeRabbit

Bug Fixes
- Corrected automatic LoRA application selection for quantized weight configurations, so they use runtime application rather than immediate application.

If runnnig LoRAs using default --lora-apply-mode auto the automatic decision between modes "immediately" and "at_runtime" works very well, but there is still a minor gap if using command line options like --type q4_0 etc.

coderabbitai · 2026-10-18T10:21:40Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI
Review profile: CHILL
Plan: Advanced
Run ID: b9035def-8edf-4c2a-8cab-617b6c7f8df7

📥 Commits

Reviewing files that changed from the base of the PR and between ee7098e and a1ce56f.

📒 Files selected for processing: 1

src/pipeline/diffusion_engine.cpp

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📝 Walkthrough

configure_weight_loading now detects quantized expected_type values in tensor storage rules. In automatic LoRA mode, this detection selects runtime application alongside the existing conditions.

Changes: Weight Loading Configuration

Layer / File(s)	Summary
Quantized tensor detection and LoRA selection `src/pipeline/diffusion_engine.cpp`	The tensor-storage scan detects quantized GGML `expected_type` values. Automatic LoRA mode selects runtime application when such a value is present.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix

Suggested reviewers: leejet

Merge Risk: ⚪ Minimal · up to a1ce5

Automatic LoRA mode now defers application when an expected type is recognized as quantized. The inspected code revealed no concrete issue requiring resolution before merge.

### Architecture Summary

Architecture risk: 🔵 Low · up to a1ce5

The change affects 1 system.

Changed systems: src

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

observed — src (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

observed — Modified behavior in src/pipeline/diffusion_engine.cpp: The tensor-storage scan now records whether any expected_type is a quantized GGML type, in addition to tracking INT8 tensorwise storage; it stops once both conditions are found.
observed — Modified behavior in src/pipeline/diffusion_engine.cpp: Automatic LoRA mode now selects runtime application when a quantized tensor-type rule is present, even if the existing quantized-weight, constrained-streaming, and row-split conditions are absent.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description check		The description explains the `--type q4_0` issue, provides reproduction examples, includes crash context, and completes the checklist. The Summary section retains its placeholder, and no related issue…
Title check		The title clearly and concisely describes the main change: improving automatic `--lora-apply-mode` decisions for quantized parameters.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Description check

Explanation

The description explains the --type q4_0 issue, provides reproduction examples, includes crash context, and completes the checklist. The Summary section retains its placeholder, and no related issue is linked, but the required information is otherwise clear.

Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Some tools did not complete. Review the errors below.

🔧 Cppcheck (2.22.0)

src/pipeline/diffusion_engine.cpp

Cppcheck timed out; analysis of this file is incomplete

_{Comment @coderabbitai help to get the list of available commands.}

leejet

Could we check each tensor’s effective type (expected_type when set, otherwise type) instead of only the global wtype?

--tensor-type-rules takes precedence over --type, but get_wtype_stat() only counts the original storage types. With a floating-point model:

--tensor-type-rules 'model.diffusion_model.*=q4_0' still selects immediately, leaving the same class of issue unresolved.
--type q4_0 --tensor-type-rules '.*=f16' now selects at_runtime, even though the resulting weights are not quantized.

Checking the effective types would cover both cases while preserving the existing INT8 tensorwise handling.

akleine · 2026-10-19T12:49:24Z

@leejet Thank you for your review!
Actually, I had never used the --tensor-type-rules option - as you might have guessed.

So here comes an update that addresses the mentioned issues.
Perhaps some parts can be shortened a bit more. (Please feel free to edit some more.)
Once again thank you for your work ❤️

leejet · 2026-10-20T14:47:57Z

feat: enhance decision in --lora-apply-mode auto

ee7098e

If runnnig LoRAs using default --lora-apply-mode auto the automatic decision between modes "immediately" and "at_runtime" works very well, but there is still a minor gap if using command line options like --type q4_0 etc.

leejet requested changes Oct 18, 2026

View reviewed changes

feat: decide LoRA mode by checking each tensor’s effective type

a1ce56f

leejet merged commit a068716 into leejet:master Oct 20, 2026
11 checks passed

leehack mentioned this pull request Oct 21, 2026

Qualify post-v0.2 upstream changes with context ABI and FP8 CPU safety checks leehack/stable-diffusion-native#20

Open

6 tasks

danielhanchen mentioned this pull request Oct 21, 2026

Merge upstream master-982 (MOSS-Image) and port the ggml patches to dc99877 unslothai/stable-diffusion.cpp#34

Merged

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

feat: enhance decision in --lora-apply-mode auto for quantized parameters - #2162

feat: enhance decision in --lora-apply-mode auto for quantized parameters#2162
leejet merged 2 commits into
leejet:masterfrom
akleine:master

akleine commented Oct 18, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 18, 2026 •

edited

Loading

Uh oh!

leejet left a comment

Uh oh!

akleine commented Oct 19, 2026

Uh oh!

Uh oh!

leejet commented Oct 20, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

akleine commented Oct 18, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue / Discussion

Additional Information

Checklist

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

leejet left a comment

Choose a reason for hiding this comment

Uh oh!

akleine commented Oct 19, 2026

Uh oh!

Uh oh!

leejet commented Oct 20, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

akleine commented Oct 18, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 18, 2026 •

edited

Loading