Preserve the sub-3B ngram fallback during partial offload

For an embedded Hybrid Mamba model below the 3B MTP threshold that genuinely requires --fit on (for example, a 2B model on a very small GPU), this flag causes _build_speculative_flags to return --spec-type none before reaching its existing _mtp_too_small branch. Auto previously substitutes ngram-mod for these models because embedded MTP is slower while ngram speculation can still help, so the new partial-offload path should defer to that policy rather than disabling all speculation.

Useful? React with 👍 / 👎.

Not taking this one. The ordering is as you describe, but ngram speculation is not the cheap substitute on a Hybrid Mamba target that it is elsewhere. common_params_speculative::need_n_rs_seq() returns draft.n_max only for DRAFT_MTP, DRAFT_EAGLE3, DRAFT_DFLASH and DRAFT_DSPARK, so every ngram type leaves n_rs_seq at 0. common_context_can_seq_rm then fails its partial seq_rm probe, since llama_memory_recurrent::seq_rm rejects a rollback above n_rs_seq, and classifies the target COMMON_CONTEXT_SEQ_RM_TYPE_FULL. In that state server-context.cpp takes a host-side recurrent-state checkpoint on every draft round. Substituting ngram-mod here would trade a measured MTP regression for an unmeasured per-step state copy, so Auto stands down instead.

Skip recurrent VRAM when KV offload is disabled

For a Hybrid Mamba launch with --no-kv-offload (or LLAMA_ARG_KV_OFFLOAD=false), llama.cpp places the cache, including recurrent state, in host memory, but this newly added term is always included in _kv_bytes; the placement callers never pass the parsed KV-offload policy into this estimator. At four slots this adds about 598.5 MiB of nonexistent GPU usage before the rollback reserve, which can unnecessarily reduce context or force a partially CPU-offloaded launch. Thread the effective KV-offload setting through the placement estimate or omit recurrent state from the GPU budget when it is disabled.

Useful? React with 👍 / 👎.

Not taking this one. _estimate_kv_cache_bytes has never modeled --no-kv-offload: the only consumer of _kv_offload_from_args is the pipelining helper, so the attention KV in every path is already budgeted as GPU-resident under -nkvo, and at 128K that term is 8,790 MiB against the 598.5 MiB you name. The new recurrent term is consistent with the rest of the estimator rather than a new gap, and it errs in the direction this code picks on purpose: over-reserving costs context, under-reserving OOMs. Threading the KV-offload policy through the estimate means touching every path and every caller, which belongs in its own change.

Reload when the draft-depth override is cleared

If Auto stood down at an explicit draft depth (for example 3) and the user resets the setting to the platform default (None, which is 2 on GPU), this condition skips the comparison because it requires the new intent value to be non-null. Apply therefore deduplicates against the partial-offload fallback and never reruns placement, even though the lower default rollback reserve may allow full offload and re-enable MTP.

Useful? React with 👍 / 👎.

Not taking this one. None on the request means "platform default, matches any backend", which is the contract test_already_in_target_state_matches_when_draft_n_max_unset pins, and the request shape cannot tell "the user cleared it" from "the field was omitted". Dropping the non-null guard makes the comparison None != (compared or 0), which is true for a backend already at the platform default, so every Apply that omits the field would relaunch a healthy server. The same guard covers engaged draft-mtp on main, so this is not specific to the stand-down.

Reprice Auto context after standing MTP down

When Auto context sizing initially finds a fully offloaded placement with MTP but a pass-through layer override later makes the effective launch partial (for example, Auto memory mode plus --gpu-layers 42), the earlier _fit_context_to_vram calculation may already have reduced effective_ctx to accommodate the draft KV and rollback reserve. This branch then removes MTP without rerunning that calculation, so the final partial, non-speculative server is launched with a context cap sized for memory it no longer allocates. Decide this stand-down before context fitting or reprice the context without the MTP reserve.

Useful? React with 👍 / 👎.

The ordering is real, the fit does run before the stand-down, but I measured the consequence and could not find it. Driving the load twice, once as head does it and once with the MTP reserve forced to zero, the launched -c is identical in every configuration I could build, including the Auto plus --gpu-layers 42 case you name:

Same for --fit on with an automatic context (4096 either way, the sub-fit floor) and with an explicit 65536 (honored either way). The reserve reaches the fit, but it is not what sets the launched context in these cases, so there is no context to give back. Repricing would mean re-running the fit after the command is assembled, which I am not doing for a cost I cannot measure.

Honor explicit GPU placement when the probe is empty

When _get_gpu_memory returns []—for example, after a failed custom Vulkan probe—a user can still select a working GPU through pass-through --device Vulkan0 and request partial offload with --gpu-layers 42. The child then has a fixed partial GPU placement, but this guard is false because neither _detected_gpus nor Studio's gpu_indices is populated, so _partially_offloads_layers is never consulted and Auto still emits draft-mtp, retaining the slow configuration this fallback is intended to prevent. Treat an effective explicit non-CPU device selection as GPU evidence here as well.

Useful? React with 👍 / 👎.

Fixed in b126194. This is the other side of the CPU-device check from the earlier round, and it was a hole: a probe that returns nothing is not evidence of no GPU when the extras point the child at one. The guard now also accepts an effective non-CPU --device over the same argv and env view _device_selection_is_cpu reads, so the two answers cannot disagree.

Covered by test_a_hand_pinned_device_is_gpu_evidence_when_the_probe_found_none (--device Vulkan0 --gpu-layers 42 with an empty probe). It fails without the change, and the two sibling guards, no GPU at all and a CPU device selection, still pass alongside it.

Describe the speculative placement rather than the final one

When Auto selected --fit on only because the MTP rollback reserve prevented full offload, embedded_mtp_partial_offload disables MTP before the child starts; llama.cpp then fits without that reserve and can place every layer on the GPU. In that VRAM boundary case this message incorrectly reports that only part of the running model is on the GPU and recommends a placement it already has. Phrase this as the placement MTP would require, or distinguish automatic-fit fallback from a fixed partial layer count.

Useful? React with 👍 / 👎.

Agreed, and fixed in 4a8e94f. The partial verdict is priced with the MTP rollback reserve still in it, so on the fit path llama.cpp can place every layer on the GPU once this branch turns MTP off, and the old wording then described a placement the load does not have while recommending the one it already has. It now says what MTP would require instead, which stays true there and at a fixed partial layer count. The source-copy test in studio/frontend/tests/spec-fallback-partial-offload-copy.test.ts now rejects unconditional final-state claims.

Require a real planner verdict before treating fit as partial

When the VRAM probe returns no devices but the user supplies only --device Vulkan0 (or an explicit gpu_ids pin), _detected_gpus is empty and the placement planner never ran; --fit on is merely the default fallback. The device pin nevertheless satisfies the GPU-evidence guard, and this argument makes that unproven fit count as partial, so Auto disables MTP even when the selected GPU is large enough for full offload. Allow the explicit device to prove that a GPU exists, but require actual planner evidence before interpreting --fit on as partial; concrete partial layer counts can still stand down MTP independently.

Useful? React with 👍 / 👎.

Agreed, and fixed in 11a495a. use_fit starts True at its declaration, every placement-planner branch is gated on a non-empty gpus, and the except path restores True having priced nothing, so an unfitted --fit on could mean nobody looked. I reproduced four routes where Auto stood MTP down with no verdict behind it: empty probe plus a hand-pinned device (a Metal Mac on --device Metal0, or Linux and Windows after a failed Vulkan probe), the same with an explicit gpu_ids pin, _select_gpus raising with _detected_gpus already populated, and a pass-through --fit on winning last after the planner had positively proved full offload.

The fix records the verdict where the planner returns it (_placement_verdict_partial) rather than inferring it from the flag. A concrete --gpu-layers count is untouched, so the reported Qwen3.8-27B case still stands MTP down. Tests in studio/backend/tests/test_mtp_partial_offload_evidence.py cover the platform and accelerator product.

Use the target layer count at the full-offload boundary

For embedded NextN models, self.n_layers is the GGUF block_count, including the MTP blocks, while llama.cpp applies --gpu-layers against hparams.n_layer(), which subtracts those blocks. Thus, for the 65-block Qwen example with one NextN block, --gpu-layers 65 already offloads all 64 target layers plus the output, but this comparison classifies it as partial and Auto unnecessarily emits --spec-type none. Compare against the effective target-layer count after subtracting nextn_predict_layers.

Useful? React with 👍 / 👎.

Not taking this one. llama.cpp applies --gpu-layers against hparams.n_layer_all, the raw GGUF block_count with the MTP blocks still in it, not against hparams.n_layer().

llama-model.cpp:1347 computes i_gpu_start = max(n_layer_all + 1 - n_gpu_layers, 0), :1348 caps act_gpu_layers at n_layer_all + 1, and :1740 expands the -1 default to hparams.n_layer_all + 1 with the comment 'plus 1 for the output layer'. n_layer() does not appear in the placement computation at all. On the 65-block Qwen example, --gpu-layers 65 gives i_gpu_start = 66 - 65 = 1, so layer 0 is assigned to cpu_dev and a real target layer stays in host RAM. That is partial offload, and 66 is the full boundary, which is what the existing requested <= n_layers comparison already encodes.

Subtracting nextn here would also move the same boundary in _offloads_every_layer, which shares it deliberately, and that one feeds the Model Memory gate: Studio would believe a NextN model was fully GPU resident and skip page-locking weights that are actually host resident.

Preserve MTP when the architecture gate forces CPU

On a ROCm launch where the kernel-coverage gate sets _arch_gate_forced_cpu, pass-through arguments such as --device ROCm0 --gpu-layers 42 still satisfy this GPU-evidence condition and are classified as partial offload. The later _arch_gate_forced_cpu branch masks every GPU from the child, so the actual launch is CPU-only, but Auto has already emitted --spec-type none instead of retaining the CPU MTP policy. Exclude forced-CPU placements from this stand-down decision.

Useful? React with 👍 / 👎.

Not taking this one, the argv case does not reach a running server. _arch_gate_forced_cpu is only set when gpu_ids is falsy and the VRAM probe came back empty, so _detected_gpus and gpu_indices are both empty and the only GPU-evidence disjunct that can fire is _spec_extras_pick_a_gpu, which needs a hand-named device.

Take that literally, --device ROCm0 --gpu-layers 42. Nothing on this path strips --device from argv: the forced-CPU branch only drops the split-mode and tensor-split flags, and every _strip_device_extra_args call on the launch path is gated on gpu_ids, which is falsy here. So the child gets --device ROCm0 while _emit_child_gpu_visibility masks every device, and llama.cpp throws at argument parsing, common/arg.cpp:1130 'invalid device: ROCm0'. The server never starts, so what --spec-type it was given does not matter.

The one variant that does yield a working CPU-only launch needs LLAMA_ARG_DEVICE exported into Studio's own environment, on a ROCm host whose build has kernels for none of its GPUs, plus a concrete --gpu-layers, and _emit_child_gpu_visibility clears that same variable a few lines later specifically so the child does not abort. The cost there is a speculative-decoding stand-down on a misconfigured environment, not a wrong placement, so I would rather leave the guard reading what the launch was actually told.