[[autodoc]] quantizers.quantization_config.NunchakuLiteQuantizationConfig

Nunchaku Lite is a quantization backend for loading prequantized checkpoints in Diffusers. Create compatible checkpoints with [diffuse-compressor](https://github.com/rootonchair/diffuse-compressor). It quantizes and exports a transformer, then packages it as a Diffusers pipeline.

Nunchaku Lite builds on the original [Nunchaku](https://github.com/nunchaku-ai/nunchaku) inference engine,

[DeepCompressor](https://github.com/nunchaku-ai/deepcompressor) quantization library, and

[SVDQuant paper](https://arxiv.org/abs/2411.05007).

The kernels package supplies the optimized CUDA kernels, which load automatically. Install it first.

pip install -U kernels

Load the prequantized pipeline with [`~DiffusionPipeline.from_pretrained`], which reads the quantization

config from `config.json`.

import torch

from diffusers import DiffusionPipeline

model_id = "rootonchair/ERNIE-Image-Turbo-nunchaku-lite-nvfp4"

pipe = DiffusionPipeline.from_pretrained(

model_id, torch_dtype=torch.bfloat16,

).to("cuda")

prompt = "A modern red armchair in a quiet studio, soft window light, realistic product photography"

image = pipe(

prompt=prompt,

height=1024,

width=1024,

num_inference_steps=8,

guidance_scale=1.0,

).images[0]

image.save("ernie-image-turbo-nunchaku-lite.png")

Nunchaku Lite supports the following quantized linear layer formats.

The CUDA kernels currently support the following NVIDIA GPU architectures:

- `sm_75` (Turing, for example RTX 2080)

- `sm_80` (Ampere, for example A100)

- `sm_86` (Ampere, for example RTX 3090 and RTX A6000)

- `sm_89` (Ada, for example RTX 4090)

- `sm_120` (Blackwell, for example RTX 5090)

`nvfp4` checkpoints require a Blackwell or newer NVIDIA GPU. On Blackwell GPUs, use PyTorch >= 2.7 with CUDA >= 12.8.

`int4` checkpoints require a Turing or newer NVIDIA GPU.

| Method | Precision | Group size | Notes |

|---|---:|---:|---|

| `svdq_w4a4` | `nvfp4` | 16 | Uses NVFP4 runtime kernels with SVDQ low-rank correction. |

| `svdq_w4a4` | `int4` | 64 | Uses INT4 W4A4 kernels with SVDQ low-rank correction. |

| `awq_w4a16` | `int4` | 64 | Uses INT4 weight-only AWQ-style kernels. |

The `config.json` file must include a [`NunchakuLiteQuantizationConfig`]. It defines the runtime

`compute_dtype` and the target modules for each Nunchaku Lite quantization method.

- `compute_dtype`: runtime dtype for floating-point buffers in quantized modules, typically `torch.bfloat16`.

- `svdq_w4a4`: SVDQ W4A4 target config with `precision`, `group_size`, `rank`, and `targets`.

- `awq_w4a16`: AWQ W4A16 target config with `precision`, `group_size`, and `targets`.

Each entry in `targets` must point to a linear layer. Diffusers swaps each `svdq_w4a4` target for an SVDQ W4A4 layer and each `awq_w4a16` target for an AWQ W4A16 layer. The example below shows the

expected shape with shortened target lists.

List each module you want to quantize under `svdq_w4a4` or `awq_w4a16`. A module can only use one method, so don't list the same target under both.

{

"_class_name": "ErnieImageTransformer2DModel",

"quantization_config": {

"quant_method": "nunchaku_lite",

"compute_dtype": "bfloat16",

"svdq_w4a4": {

"precision": "nvfp4",

"group_size": 16,

"rank": 32,

"targets": ["layers.0.self_attention.to_q"]

},

"awq_w4a16": {

"precision": "int4",

"group_size": 64,

"targets": ["final_linear"]

}

}

}

Nunchaku Lite kernels and quantized linear layers are compatible with [`torch.compile`](../optimization/fp16#torchcompile).

Compile the quantized transformer after loading the pipeline for faster inference.

pipe.transformer = torch.compile(pipe.transformer, mode="default", fullgraph=True)

The compiled Nunchaku Lite NVFP4 pipeline runs 1.8x faster than the original BF16 pipeline (2.271s → 1.675s on an RTX PRO 6000).

- [diffuse-compressor](https://github.com/rootonchair/diffuse-compressor)

- [Nunchaku installation requirements](https://nunchaku.tech/docs/nunchaku/installation/installation.html)

Some quantization backends support loading checkpoints that were already quantized and uploaded to the Hub. In this case, the quantization configuration is stored in the model's `config.json` and is read automatically — you do not need to create a [`~quantizers.PipelineQuantizationConfig`] or pass a `quantization_config` argument.

import torch

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(

"rootonchair/ERNIE-Image-Turbo-nunchaku-lite-nvfp4",

torch_dtype=torch.bfloat16,

).to("cuda")

The following backends support loading prequantized checkpoints out of the box.

| Backend | Notes |

|---|---|

| [bitsandbytes](./bitsandbytes) | Config is saved in `config.json`; no extra arguments needed. |

| [GGUF](./gguf) | Uses `from_single_file` with Model classes; pipeline-level loading is not supported. |

| [AutoRound](./autoround) | Only loading is supported; quantize first with the AutoRound CLI or Python API. |

| [Nunchaku Lite](./nunchaku) | Config is saved in `config.json`; requires the `kernels` package. |

| [ModelOpt](./modelopt) | Supports both quantizing on the fly and loading prequantized models. |

try:

if not is_torch_available():

raise OptionalDependencyNotAvailable()

except OptionalDependencyNotAvailable:

from .utils import dummy_nunchaku_lite_objects

_import_structure["utils.dummy_nunchaku_lite_objects"] = [

name for name in dir(dummy_nunchaku_lite_objects) if not name.startswith("_")

]

else:

_import_structure["quantizers.quantization_config"].append("NunchakuLiteQuantizationConfig")

try:

if not is_torch_available():

raise OptionalDependencyNotAvailable()

except OptionalDependencyNotAvailable:

from .utils.dummy_nunchaku_lite_objects import *

else:

from .quantizers.quantization_config import NunchakuLiteQuantizationConfig

from .nunchaku import NunchakuLiteQuantizer

NunchakuLiteQuantizationConfig,

"nunchaku_lite": NunchakuLiteQuantizer,

"nunchaku_lite": NunchakuLiteQuantizationConfig,

from .nunchaku_quantizer import NunchakuLiteQuantizer

from __future__ import annotations

from typing import TYPE_CHECKING, Any

from ..base import DiffusersQuantizer

if TYPE_CHECKING:

from ...models.modeling_utils import ModelMixin

from ...utils import is_kernels_available, logging

logger = logging.get_logger(__name__)

class NunchakuLiteQuantizer(DiffusersQuantizer):

def __init__(self, quantization_config, **kwargs):

super().__init__(quantization_config, **kwargs)

self.compute_dtype = quantization_config.compute_dtype

self.pre_quantized = quantization_config.pre_quantized

def validate_environment(self, *args, **kwargs):

if not is_kernels_available():

raise ImportError(

"Loading Nunchaku checkpoints requires the Hugging Face `kernels` package. "

"Install it with `pip install kernels`."

)

import torch

cuda_available = torch.cuda.is_available()

if not cuda_available:

raise ValueError("Loading Nunchaku checkpoints requires a CUDA-capable NVIDIA GPU.")

device_capability = torch.cuda.get_device_capability()

if device_capability[0] == 9:

raise ValueError("Loading Nunchaku checkpoints is not supported on Hopper NVIDIA GPUs.")

has_nvfp4_config = (

self.quantization_config.svdq_w4a4 is not None

and self.quantization_config.svdq_w4a4["precision"] == "nvfp4"

)

has_int4_config = any(

config is not None and config["precision"] == "int4"

for config in (self.quantization_config.svdq_w4a4, self.quantization_config.awq_w4a16)

)

if has_nvfp4_config and device_capability < (10, 0):

raise ValueError("Loading Nunchaku NVFP4 checkpoints requires a Blackwell or newer NVIDIA GPU.")

if has_int4_config and device_capability < (7, 5):

raise ValueError("Loading Nunchaku INT4 checkpoints on CUDA requires a Turing or newer NVIDIA GPU.")

def update_torch_dtype(self, torch_dtype):

if torch_dtype is None:

torch_dtype = self.compute_dtype

else:

self.compute_dtype = torch_dtype

return torch_dtype

def _process_model_before_weight_loading(

self,

model: "ModelMixin",

state_dict: dict[str, Any] | None = None,

**kwargs,

):

from .utils import check_strict_state_dict_match, replace_with_nunchaku_linear

quantization_config = self.quantization_config.to_dict()

num_replaced = replace_with_nunchaku_linear(model, quantization_config, self.compute_dtype)

if state_dict is not None:

check_strict_state_dict_match(model, state_dict)

logger.info(f"Applied Nunchaku quantization config with {num_replaced} targets.")

def _process_model_after_weight_loading(self, model: "ModelMixin", **kwargs):

return model

@property

def is_serializable(self):

return False

@property

def is_trainable(self) -> bool:

return False

@property

def is_compileable(self) -> bool:

return True