iframe-proxy | Sunbelt Computer Software

SGLang supports various quantization methods, including offline quantization and online dynamic quantization. Offline quantization loads pre-quantized model weights directly during inference. This is required for quantization methods such as GPTQ and AWQ, which collect and pre-compute various statistics from the original weights using the calibration dataset. Online quantization dynamically computes scaling parameters—such as the maximum/minimum values of model weights—during runtime. Like NVIDIA FP8 training’s delayed scaling mechanism, online quantization calculates the appropriate scaling factors on-the-fly to convert high-precision weights into a lower-precision format. Note: For better performance, usability and convenience, offline quantization is recommended over online quantization. If you use a pre-quantized model, do not add --quantization to enable online quantization at the same time. For popular pre-quantized models, please visit Unsloth, NVIDIA ModelOpt or NeuralMagic collections on HF for some popular quality validated quantized models. Quantized models must be validated via benchmarks post-quantization to guard against abnormal quantization loss regressions.

Platform Compatibility

The following table summarizes quantization method support across NVIDIA and AMD GPUs, Ascend NPUs.

Method	NVIDIA GPUs	AMD GPUs (MI300X/MI325X/MI350X)	Ascend NPUs (A2/A3 Series, 950PR/DT Series)	Notes
`fp8`	Yes	Yes	WIP	Aiter or Triton backend on AMD
`mxfp4`	Yes	Yes	Yes (950PR/DT Series)	On GPU: requires CDNA3/CDNA4 with MXFP support (uses Aiter). On Ascend NPU (950PR/DT Series): W4A4 MXFP4 for Qwen3 dense and MoE LLMs (MXFP4 weights + activations) — dense models support online dual-level MXFP4; offline `W4A4_MXFP4` dense and MoE checkpoints (single-level) are auto-detected via `modelslim`. On Intel GPUs (XPU): W4A16 MoE experts on Xe2/BMG via `sgl-kernel-xpu`, enabled automatically with `—device xpu` (see Intel GPUs (XPU))
`mxfp8`	No	No	Yes (950PR/DT Series for Diffusion, LLM Dense Linear and LLM MoE)	Ascend NPU only; online + offline MXFP8 for Diffusion models (e.g., Wan2.2), LLM Dense Linear, and LLM MoE (FusedMoE, e.g. Qwen3-30B-A3B) on 950PR/DT Series; uses CANN `npu_dynamic_mx_quant` / `npu_quant_matmul` (dense) and `npu_grouped_matmul_swiglu_quant_v2` / `npu_grouped_matmul` (MoE) kernels
`mxfp_w4a8`	No	No	Yes (950PR/DT Series)	Ascend NPU only; online W4A8 for Qwen3 dense LLM (MXFP4 weights + MXFP8 activations) on 950PR/DT Series; offline `W4A8_MXFP` dense and MoE checkpoints are auto-detected via `modelslim`
`blockwise_int8`	Yes	Yes	No	Triton-based, works on both platforms
`w8a8_int8`	Yes	Yes	No
`w8a8_fp8`	Yes	Yes	No	Aiter or Triton FP8 on AMD
`awq`	Yes	Yes	Yes	Uses a JIT-compiled CUDA kernel on NVIDIA, Triton dequantize on AMD. Uses CANN kernels on Ascend
`gptq`	No	No	Yes	Removed on NVIDIA and AMD GPUs — use `gptq_marlin` instead. Uses CANN kernels on Ascend. Still supported on Intel CPUs with AMX
`compressed-tensors`	Yes	Yes	Partial	Aiter paths for FP8/MoE on AMD. Uses CANN kernels on Ascend, `FP8` not supported yet
`quark`	Yes	Yes	No	AMD Quark quantization; Aiter GEMM paths on AMD
`auto-round`	Yes	Yes	Partial	Platform-agnostic (Intel auto-round). Uses CANN kernels on Ascend
`quark_int4fp8_moe`	No	Yes	No	AMD-only; online INT4-to-FP8 MoE quantization (CDNA3/CDNA4)
`awq_marlin`	Yes	No	No	Marlin kernels are CUDA-only
`gptq_marlin`	Yes	No	No	Marlin kernels are CUDA-only
`gguf`	Yes	No	Yes	CUDA kernels in sgl-kernel; Ascend uses CPU pre-dequantization at load time
`modelopt` / `modelopt_fp8`	Yes (Hopper/SM90+)	No	No	NVIDIA ModelOpt; requires NVIDIA hardware
`modelopt_fp4`	Yes (SM80-SM90 via Marlin; SM100+ native FP4)	No	No	NVIDIA ModelOpt; use Marlin W4A16 fallback on Ampere/Hopper and native FP4 backends on Blackwell; supports load-time BF16/FP16/FP8 MoE conversion with per-tensor FP32 activation scales
`nvfp4_online`	Yes (Blackwell/SM100 or SM103)	No	No	Online MoE-only NVFP4 weight quantization with per-token FP32 activation scales for BF16/FP16/FP8 checkpoints; use `modelopt_fp4` for per-tensor FP32 activation scales; requires `flashinfer_trtllm` or `flashinfer_trtllm_routed`
`petit_nvfp4`	No	Yes (MI250/MI300X/MI325X)	No	Enables NVFP4 on ROCm via Petit; use `modelopt_fp4` on NVIDIA Blackwell. Auto-selected when loading NVFP4 models on AMD. See LMSYS blog and AMD ROCm blog.
`bitsandbytes`	Yes	Experimental	No	Depends on bitsandbytes ROCm support
`modelslim`	No	No	Yes	Ascend quantization; Uses CANN kernels

On AMD, several of these methods use Aiter for acceleration — set SGLANG_USE_AITER=1 where noted. See AMD GPU setup for installation and configuration details. On Ascend, various layers quantization configurations are supported, see Ascend NPU quantization for details.

GEMM Backends for FP4/FP8 Quantization

Backend selection applies to blockwise FP8, MXFP8 (dense linear), and NVFP4 GEMM. When running offline or online FP8 or FP4 quantized models, you can select the GEMM backend via --fp8-gemm-backend and --fp4-gemm-backend.

`--fp8-gemm-backend` (Blockwise FP8 GEMM)

Backend	Hardware	Description
`auto`	All	Auto-selects based on hardware
`deep_gemm`	SM90, SM100	JIT-compiled; enabled when DeepGEMM is installed
`flashinfer_trtllm`	SM100	FlashInfer TensorRT-LLM backend; optimal for low-latency
`flashinfer_cutlass`	SM100/120	FlashInfer CUTLASS groupwise FP8 GEMM
`flashinfer_deepgemm`	SM90	Uses swapAB optimization for small M dimensions in decoding
`cutlass`	SM120	sgl-kernel CUTLASS
`triton`	All	Fallback; widely compatible
`aiter`	ROCm	AMD AITER backend

auto selection order: 1) DeepGEMM (SM90/SM100, installed); 2) FlashInfer TRTLLM (SM100, FlashInfer available); 3) CUTLASS (SM120); 4) AITER (AMD); 5) Triton (fallback). MXFP8 dense linear: auto uses flashinfer_cutlass on SM100 (else triton). flashinfer_cutlass is fastest on most shapes; flashinfer_trtllm is faster only at small M.

`--fp4-gemm-backend` (NVFP4 GEMM)

Backend	Hardware	Description
`auto`	SM80+	Auto-selects: `flashinfer_cutedsl` on SM100; `marlin` on SM80-SM90; `flashinfer_cutlass` otherwise (including SM120)
`flashinfer_cutlass`	SM100/120	FlashInfer CUTLASS backend
`flashinfer_cudnn`	SM100/120 (CUDA 13+, cuDNN 9.15+)	FlashInfer cuDNN backend
`flashinfer_cutedsl`	SM100	FlashInfer CuTe DSL backend
`flashinfer_trtllm`	SM100	FlashInfer TensorRT-LLM backend
`marlin`	SM80-SM90	Weight-only W4A16 fallback for NVFP4 checkpoints

On SM80-SM90, auto selects Marlin for NVFP4. NVFP4 GEMM requires FlashInfer to be installed.

Offline Quantization

To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there’s no need to add --quantization argument when starting the engine. The quantization method will be parsed from the downloaded Hugging Face or msModelSlim config. For example, DeepSeek V3/R1 models are already in FP8, so do not add redundant parameters.

Command

python3 -m sglang.launch_server \
    --model-path hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 \
    --port 30000 --host 0.0.0.0

Take note, if your model is per-channel quantized (INT8 or FP8) with per-token dynamic quantization activation, you can opt to include --quantization w8a8_int8 or --quantization w8a8_fp8 to invoke the corresponding CUTLASS int8_kernel or fp8_kernel in sgl-kernel. This action will ignore the Hugging Face config’s quantization settings. For instance, with neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic, if you execute with --quantization w8a8_fp8, the system will use the W8A8Fp8Config from SGLang to invoke the sgl-kernel, rather than the CompressedTensorsConfig for vLLM kernels.

Command

python3 -m sglang.launch_server \
    --model-path neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic \
    --quantization w8a8_fp8 \
    --port 30000 --host 0.0.0.0

Examples of Offline Model Quantization

Using Unsloth

We strongly suggest the use of Unsloth to quantize and load the model. Please refer to SGLang Deployment & Inference Guide with Unsloth.

Using auto-round

Command

# Install
pip install auto-round

LLM quantization

Example

# for LLM
from auto_round import AutoRound
model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-autoround-4bit"
# Scheme examples: "W2A16", "W3A16", "W4A16", "W8A16", "NVFP4", "MXFP4" (no real kernels), "GGUF:Q4_K_M", etc.
scheme = "W4A16"
format = "auto_round"
autoround = AutoRound(model_id, scheme=scheme)
autoround.quantize_and_save(quant_path, format=format) # quantize and save

VLM quantization

Example

# for VLMs
from auto_round import AutoRoundMLLM
model_name = "Qwen/Qwen2-VL-2B-Instruct"
quant_path = "Qwen2-VL-2B-Instruct-autoround-4bit"
scheme = "W4A16"
format = "auto_round"
autoround = AutoRoundMLLM(model_name, scheme)
autoround.quantize_and_save(quant_path, format=format) # quantize and save

Command Line Usage (Gaudi/CPU/Intel GPU/CUDA)

Command

auto-round \
    --model meta-llama/Llama-3.2-1B-Instruct \
    --bits 4 \
    --group_size 128 \
    --format "auto_round" \
    --output_dir ./tmp_autoround

SGlang API Usage (CPU/CUDA)

Example

from sglang.srt.configs.load_config import LoadConfig
from sglang.srt.configs.model_config import ModelConfig
from sglang.srt.model_loader.loader import get_model_loader
from sglang.srt.configs.device_config import DeviceConfig

# Configure model with inc quantization and saving
model_config = ModelConfig(
    model_path="meta-llama/Llama-3.2-3B-Instruct",
    quantization="auto-round-int8",
    trust_remote_code=True,
)

load_config = LoadConfig(
    inc_save_path="./quantized_model",
)
device_config = DeviceConfig(device="cpu")

# Load and quantize the model
model_loader = get_model_loader(load_config, model_config)
quantized_model = model_loader.load_model(
    model_config=model_config,
    device_config=device_config,
)

known issues

Several limitations currently affect offline quantized model loading in sglang, These issues might be resolved in future updates of sglang. If you experience any problems, consider using Hugging Face Transformers as an alternative.

Mixed-bit Quantization Limitations Mixed-bit quantization is not fully supported. Due to vLLM’s layer fusion (e.g., QKV fusion), applying different bit-widths to components within the same fused layer can lead to compatibility issues.
Limited Support for Quantized MoE Models Quantized MoE models may encounter inference issues due to kernel limitations (e.g., lack of support for mlp.gate layer quantization). please try to skip quantizing these layers to avoid such errors.
Limited Support for Quantized VLMs
Details
Qwen2.5-VL-7Bauto_round:auto_gptq format: Accuracy is close to zero.GPTQ format: Fails with:
Output
The output size is not aligned with the quantized weight shape
auto_round:auto_awq and AWQ format: These work as expected.
Limited Support for SGlang API Usage SGlang API Usage only supports auto-round-int8 quantization method now, more quantization methods are on the way.

CPU serving AutoRound INT4 checkpoints (both auto_round:auto_gptq and auto_round:auto_awq packing formats) can be served on Intel CPUs with AMX support:
SGLANG_USE_CPU_ENGINE=1 python3 -m sglang.launch_server \ --model-path OPEA/Qwen2.5-0.5B-Instruct-int4-sym-inc \ --quantization auto-round \ --device cpu --trust-remote-code
The current SGLang CPU backend supports only 4-bit AutoRound checkpoints on Intel AMX. Other AutoRound bit-widths and non-AMX CPU backends are not supported by this path.

Using GPTQModel

Command

# install
pip install gptqmodel --no-build-isolation -v

Example

from datasets import load_dataset
from gptqmodel import GPTQModel, QuantizeConfig

model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-gptqmodel-4bit"

calibration_dataset = load_dataset(
    "allenai/c4", data_files="en/c4-train.00001-of-01024.json.gz",
    split="train"
  ).select(range(1024))["text"]

quant_config = QuantizeConfig(bits=4, group_size=128) # quantization config
model = GPTQModel.load(model_id, quant_config) # load model

model.quantize(calibration_dataset, batch_size=2) # quantize
model.save(quant_path) # save model

Using LLM Compressor

Command

# install
pip install llmcompressor

Here, we take quantize meta-llama/Meta-Llama-3-8B-Instruct to FP8 as an example to elaborate on how to do offline quantization.

Example

from transformers import AutoTokenizer
from llmcompressor.transformers import SparseAutoModelForCausalLM
from llmcompressor.transformers import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

# Step 1: Load the original model.
MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"

model = SparseAutoModelForCausalLM.from_pretrained(
  MODEL_ID, device_map="auto", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

# Step 2: Perform offline quantization.
# Step 2.1: Configure the simple PTQ quantization.
recipe = QuantizationModifier(
  targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])

# Step 2.2: Apply the quantization algorithm.
oneshot(model=model, recipe=recipe)

# Step 3: Save the model.
SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic"
model.save_pretrained(SAVE_DIR)
tokenizer.save_pretrained(SAVE_DIR)

Then, you can directly use the quantized model with SGLang, by using the following command:

Command

python3 -m sglang.launch_server \
    --model-path $PWD/Meta-Llama-3-8B-Instruct-FP8-Dynamic \
    --port 30000 --host 0.0.0.0

Using NVIDIA ModelOpt

NVIDIA Model Optimizer (ModelOpt) provides advanced quantization techniques optimized for NVIDIA hardware. Offline vs. Online Quantization: SGLang supports two modes for ModelOpt.

Offline Quantization (pre-quantized):
- Usage: Download a pre-quantized model from Hugging Face or run hf_ptq.py once to create a new quantized checkpoint. Then load this quantized checkpoint.
- Pros: Fast server startup, quantization can be validated before deployment, efficient resource usage.
- Cons: Requires an extra preparation step.
Online Quantization (quant and serve):
- Usage: Load a standard BF16/FP16 model and add a flag. The engine applies quantization on startup.
- Pros: Convenient (no new checkpoint needed).
- Cons: High startup time, increases VRAM usage during initialization (risk of OOM).

The following sections guide you through using the Offline path: loading pre-quantized models or creating your own checkpoints.

Using Pre-Quantized Checkpoints

If a model is already quantized (e.g., from Hugging Face), you can load it directly.

FP8 Models: Use --quantization modelopt_fp8.

Command

python3 -m sglang.launch_server \
    --model-path nvidia/Llama-3.1-8B-Instruct-FP8 \
    --quantization modelopt_fp8 \
    --port 30000

FP4 Models: Use --quantization modelopt_fp4.

Command

python3 -m sglang.launch_server \
    --model-path nvidia/Llama-3.3-70B-Instruct-NVFP4 \
    --quantization modelopt_fp4 \
    --port 30000

Creating Your Own Quantized Checkpoints

If a pre-quantized checkpoint is not available for your model, you can create one using NVIDIA Model Optimizer’s hf_ptq.py script. Why quantize?

Reduce VRAM usage
Higher throughput and lower latency
More flexible deployment (on smaller GPUs)

What can be quantized?

The entire model
MLP layers only
KV cache

Key options in hf_ptq.py: --qformat: Quantization formats fp8, nvfp4, nvfp4_mlp_only --kv_cache_qformat: KV cache quantization format (default: fp8) Note: The default kv_cache_qformat may not be optimal for all use cases. Consider setting this explicitly. Hardware requirements: Hopper and higher are recommended. Insufficient GPU memory may cause weight offloading, resulting in extremely long quantization time. For detailed usage and supported model architectures, see NVIDIA Model Optimizer LLM PTQ. SGLang includes a streamlined workflow for quantizing models with ModelOpt and automatically exporting them for deployment.

Installation

First, install ModelOpt:

Command

pip install nvidia-modelopt

Quantization and Export Workflow

SGLang provides an example script that demonstrates the complete ModelOpt quantization and export workflow. Run from the SGLang repository root (see modelopt_quantize_and_export.py):

Command

# Quantize and export a model using ModelOpt FP8 quantization
python examples/usage/modelopt_quantize_and_export.py quantize \
    --model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
    --export-dir ./quantized_tinyllama_fp8 \
    --quantization-method modelopt_fp8

# For FP4 quantization (requires Blackwell GPU)
python examples/usage/modelopt_quantize_and_export.py quantize \
    --model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
    --export-dir ./quantized_tinyllama_fp4 \
    --quantization-method modelopt_fp4

Available Quantization Methods

modelopt_fp8: FP8 quantization with optimal performance on NVIDIA Hopper and Blackwell GPUs
modelopt_fp4: FP4 quantization with optimal performance on Nvidia Blackwell GPUs

Python API Usage

You can also use ModelOpt quantization programmatically:

Example

import sglang as sgl
from sglang.srt.configs.device_config import DeviceConfig
from sglang.srt.configs.load_config import LoadConfig
from sglang.srt.configs.model_config import ModelConfig
from sglang.srt.model_loader.loader import get_model_loader

# Configure model with ModelOpt quantization and export
model_config = ModelConfig(
    model_path="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    quantization="modelopt_fp8",  # or "modelopt_fp4"
    trust_remote_code=True,
)

load_config = LoadConfig(
    modelopt_export_path="./exported_model",
    modelopt_checkpoint_save_path="./checkpoint.pth",  # optional, fake quantized checkpoint
)
device_config = DeviceConfig(device="cuda")

# Load and quantize the model (export happens automatically)
model_loader = get_model_loader(load_config, model_config)
quantized_model = model_loader.load_model(
    model_config=model_config,
    device_config=device_config,
)

Deploying Quantized Models

After quantization and export, you can deploy the model with SGLang:

Command

# Deploy the exported quantized model
python -m sglang.launch_server \
    --model-path ./quantized_tinyllama_fp8 \
    --quantization modelopt \
    --port 30000 --host 0.0.0.0

Or using the Python API (use the same path as modelopt_export_path from the quantize step):

Example

import sglang as sgl

def main():
    # Deploy exported ModelOpt quantized model
    # Path must match modelopt_export_path from quantize step (e.g., ./exported_model)
    llm = sgl.Engine(
        model_path="./exported_model",
        quantization="modelopt",
    )

    # Run inference
    prompts = [
        "Hello, how are you?",
        "What is the capital of France?",
    ]
    sampling_params = {
        "temperature": 0.8,
        "top_p": 0.95,
        "max_new_tokens": 100,
    }

    outputs = llm.generate(prompts, sampling_params)

    for i, output in enumerate(outputs):
        print(f"Prompt: {prompts[i]}")
        print(f"Output: {output['text']}")

if __name__ == "__main__":
    main()

Advanced Features

Checkpoint Management: Save and restore fake quantized checkpoints for reuse:

Command

# Save the fake quantized checkpoint during quantization
python examples/usage/modelopt_quantize_and_export.py quantize \
    --model-path meta-llama/Llama-3.2-1B-Instruct \
    --export-dir ./quantized_model \
    --quantization-method modelopt_fp8 \
    --checkpoint-save-path ./my_checkpoint.pth

# The checkpoint can be reused for future quantization runs and skip calibration

Export-only Workflow: If you have a pre-existing fake quantized ModelOpt checkpoint, you can export it directly. See LoadConfig for the full API:

Example

from sglang.srt.configs.device_config import DeviceConfig
from sglang.srt.configs.load_config import LoadConfig
from sglang.srt.configs.model_config import ModelConfig
from sglang.srt.model_loader.loader import get_model_loader

model_config = ModelConfig(
    model_path="meta-llama/Llama-3.2-1B-Instruct",
    quantization="modelopt_fp8",
    trust_remote_code=True,
)

load_config = LoadConfig(
    modelopt_checkpoint_restore_path="./my_checkpoint.pth",
    modelopt_export_path="./exported_model",
)

# Load and export the model (DeviceConfig defaults to device="cuda")
model_loader = get_model_loader(load_config, model_config)
model_loader.load_model(model_config=model_config, device_config=DeviceConfig())

Benefits of ModelOpt

Hardware Optimization: Specifically optimized for NVIDIA GPU architectures
Advanced Quantization: Supports cutting-edge FP8 and FP4 quantization techniques
Seamless Integration: Automatic export to HuggingFace format for easy deployment
Calibration-based: Uses calibration datasets for optimal quantization quality
Production Ready: Enterprise-grade quantization with NVIDIA support

Using ModelSlim

MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression tool launched by MindStudio and optimized for Ascend hardware.

Installation

Command

# Clone repo and install msmodelslim:
git clone https://gitcode.com/Ascend/msmodelslim.git
cd msmodelslim
bash install.sh

LLM quantization Download the original floating-point weights of the large model. Taking Qwen3-32B as an example, you can go to Qwen3-32B to obtain the original model weights. Then install other dependencies (related to the model, refer to the huggingface model card).

Note: You can find pre-quantized validated models on modelscope/Eco-Tech.

Traditional quantification methods require the preparation of calibration data files (.jsonl formats) for calibration in the quantification process.

Command

Qwen3-32B/      # floating-point model downloaded from official HF (or modelscope) repo
msmodelslim/    # msmodelslim repo
  |----- lab_calib # calibration date folder (put your dataset here in ```.jsonl``` format or use pre-prepared ones)
      |----- some file (such as laos_calib.jsonl)
  |----- lab_practice # best practice folder with configs for quantization
      |----- model folder (such as qwen3_5_moe folder) # folder with quantization configs
          |----- quant_config (such as qwen3_5_moe_w8a8.yaml) # quantization config
  |----- another folders
output_folder/   # generated by below command
  |----- quant_model_weights-00001-of-0001.safetensors # quantized weights
  |----- quant_model_description.json # file with description of the quantization methods for each layer (```W4A4_DYNAMIC```, etc.)
  |----- another files (such as config.json, tokenizer.json, etc.)

Run quantization using one-click quantization (recommended):

Command

msmodelslim quant \
--model_path ${MODEL_PATH} \
--save_path ${SAVE_PATH} \
--device npu:0,1 \
--model_type Qwen3-32B \
--quant_type w8a8 \
--trust_remote_code True

Usage Example

Command

python3 -m sglang.launch_server \
--model-path $PWD/Qwen3-32B-w8a8 \
--port 30000 --host 0.0.0.0

Available Quantization Methods:
- W4A4_DYNAMIC linear with online quantization of activations
- W8A8 linear with offline quantization of activations
- W8A8_DYNAMIC linear with online quantization of activations
- W4A4_DYNAMIC MOE with online quantization of activations
- W4A4_MXFP4 MOE with dynamic MXFP4 activation quantization
- W4A8_DYNAMIC MOE with online quantization of activations
- W4A8_MXFP MOE with dynamic MXFP8 activation quantization
- W8A8_DYNAMIC MOE with online quantization of activations
- W4A8 linear TBD
- W4A16 linear TBD
- W48A16 linear TBD
- W4A16 MoE in progress
- W8A16 MoE in progress
- KV Cache in progress
- Attention in progress

For more detailed examples of quantization of models, as well as information about their support, see the examples section in ModelSLim repo.

Online Quantization

To enable online quantization, you can simply specify --quantization in the command line. For example, you can launch the server with the following command to enable FP8 quantization for model meta-llama/Meta-Llama-3.1-8B-Instruct:

Command

python3 -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --quantization fp8 \
    --port 30000 --host 0.0.0.0

Our team is working on supporting more online quantization methods. SGLang will soon support methods including but not limited to ["awq", "gptq", "marlin", "gptq_marlin", "awq_marlin", "bitsandbytes", "gguf"].

`nvfp4_online` online quantization method

Use --quantization nvfp4_online to convert eligible BF16, FP16, or FP8 MoE expert weights to NVFP4 at load time with per-token FP32 activation scales. Use modelopt_fp4 for serialized NVFP4 checkpoints or the same load-time conversion with per-tensor FP32 activation scales. The design separates weight quantization from activation scaling:

Weights: SGLang quantizes each eligible MoE expert weight tensor as it is loaded, using standard 2D NVFP4 weight quantization. The generated NVFP4 weights use static E4M3 block scales plus static per-tensor FP32 scales derived from the weight amax. For gated MoE experts, the w1/w3 pair shares one per-tensor FP32 scale.
Activations: FlashInfer computes and propagates one FP32 scale per token at runtime. Backends that accept one per-tensor FP32 activation scale must use modelopt_fp4.
FP8 checkpoints: If an eligible expert weight is stored as FP8, SGLang first dequantizes that tensor with the checkpoint scale and then requantizes it to NVFP4 during loading.
Other layers: Dense linear layers stay in their source checkpoint precision or checkpoint quantization path.

Only --moe-runner-backend flashinfer_trtllm and --moe-runner-backend flashinfer_trtllm_routed are supported. If --moe-runner-backend is omitted, SGLang selects flashinfer_trtllm. Tensor parallelism is supported; activation per-token scales are computed locally on each TP rank, while online weight quantization still uses the loaded expert tensor’s per-tensor amax-derived FP32 scale. FlashInfer TRTLLM MoE backends disable shared-expert fusion, so online quantization applies to routed MoE experts while shared experts stay in the checkpoint precision. Both online modes honor SGLANG_FP4_IGNORED_LAYERS; for FP8 source checkpoints, listed experts remain FP8 instead of being converted to NVFP4.

Command

python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 \
    --tp-size 2 \
    --ep-size 2 \
    --quantization nvfp4_online \
    --port 30000 --host 0.0.0.0

`quark_int4fp8_moe` online quantization method

SGLang running on AMD GPUs (CDNA3 or CDNA4 architecture) supports the quantization method --quantization quark_int4fp8_moe, that will replace MoE layers originally in high precision (bfloat16, float16 or float32) to use weights dynamically quantized to int4, that are upcasted to float8 during inference to run compute in float8 precision with activations dynamically quantized on the fly to float8. Other layers (e.g. projections in the attention layers) have their weights quantized online to float8 directly.

`quark_mxfp4` online quantization method

SGLang running on AMD GPUs with hardware FP4 support (CDNA4 architecture, e.g. MI355x) supports the quantization method --quantization quark_mxfp4, that will quantize BF16 or NVFP4 model weights to MXFP4 at load time, use dynamic MXFP4 quantization for activations and MXFP4 GEMMs instead of BF16 GEMMs. Example (BF16 to MXFP4 requantization):

sglang serve --model-path Qwen/Qwen3-30B-A3B \
    --tensor-parallel-size 1 \
    --quantization quark_mxfp4

Online NVFP4 to MXFP4 Requantization

The option --quantization quark_mxfp4 supports converting NVFP4 checkpoints (e.g. nvidia/Kimi-K2.6-NVFP4) to MXFP4 at load time to allow efficient inference using supported AMD hardware (gfx95x+):

The quantization metadata of the source NVFP4 checkpoint is read from either config.json (quantization_config) or a standalone hf_quant_config.json;
Producer-declared excluded modules will remain in higher precision;
NVFP4 checkpoints with mixed precision ("quant_algo": "MIXED_PRECISION", e.g. nvidia/Qwen3.5-397B-A17B-NVFP4-V2) are also supported.

Example (NVFP4 to MXFP4 requantization):

sglang serve --model-path nvidia/Kimi-K2.6-NVFP4 \
    --tensor-parallel-size 4 \
    --quantization quark_mxfp4 \

Online FP8 to MXFP4 Requantization

The option --quantization quark_mxfp4 supports converting FP8 dense and MOE models to MXFP4, following this logic:

Load an FP8 weight tensor,
Dequantize it to BF16,
Requantize it to MXFP4

progressively during weight loading. Example (FP8 to MXFP4 requantization):

sglang serve --model-path Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \
    --tensor-parallel-size 1 \
    --quantization quark_mxfp4

Intel® Neural Compressor online quantization method

SGLang supports quantization methods based on the advanced algorithm auto-round in Intel® Neural Compressor. You can simply specify --quantization auto-round-int8 to use this feature. It will quantize the model on the fly to target format. More online quantization methods are on the way.

Available Quantization Methods

Quantization Method	Schemes	Validated Hardware Environment
auto-round-int8	INT8 per-channel quantized weight INT8 per-token dynamic quantized activation	Intel Xeon Scalable processor Nvidia A100 GPU

Diffusion Model Quantization on Ascend NPU

SGLang-Diffusion supports MXFP8 quantization for diffusion models (such as Wan2.2) on Ascend 950PR/DT Series NPUs, in both online and offline (ModelSlim) modes. This is separate from the LLM serving path and uses the sglang serve / sglang generate CLI. Requirements: Ascend 950PR/DT Series, CANN ≥ 8.0.RC3

Online MXFP8

Pass --quantization mxfp8 to dynamically quantize FP16/BF16 transformer weights to MXFP8 at load time:

sglang serve \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp8 \
  --num-gpus 4

Offline MXFP8 (ModelSlim)

Pre-quantize with msModelSlim and load the checkpoint directly — the quantization scheme is auto-detected from quant_model_description.json:

sglang generate \
  --model-path /path/to/wan2_2_mxfp8_diffusers \
  --prompt "a beautiful sunset" \
  --save-output

For the full quantization + format conversion workflow and a complete list of supported schemes, see Diffusion Quantization on Ascend NPU and SGLang-Diffusion Quantization.

Sunbelt Computer Software

PL/B Language Development and Support

​Platform Compatibility

​GEMM Backends for FP4/FP8 Quantization

​--fp8-gemm-backend (Blockwise FP8 GEMM)

​--fp4-gemm-backend (NVFP4 GEMM)

​Offline Quantization

​Examples of Offline Model Quantization

​Using Unsloth

​Using auto-round

​Using GPTQModel

​Using LLM Compressor

​Using NVIDIA ModelOpt

Using Pre-Quantized Checkpoints

Creating Your Own Quantized Checkpoints

Installation

Quantization and Export Workflow

Available Quantization Methods

Python API Usage

Deploying Quantized Models

Advanced Features

Benefits of ModelOpt

​Using ModelSlim

​Online Quantization

​nvfp4_online online quantization method

​quark_int4fp8_moe online quantization method

​quark_mxfp4 online quantization method

​Online NVFP4 to MXFP4 Requantization

​Online FP8 to MXFP4 Requantization

​Intel® Neural Compressor online quantization method

Available Quantization Methods

​Diffusion Model Quantization on Ascend NPU

​Online MXFP8

​Offline MXFP8 (ModelSlim)

​Reference

Platform Compatibility

GEMM Backends for FP4/FP8 Quantization

`--fp8-gemm-backend` (Blockwise FP8 GEMM)

`--fp4-gemm-backend` (NVFP4 GEMM)

Offline Quantization

Examples of Offline Model Quantization

Using Unsloth

Using auto-round

Using GPTQModel

Using LLM Compressor

Using NVIDIA ModelOpt

Using ModelSlim

Online Quantization

`nvfp4_online` online quantization method

`quark_int4fp8_moe` online quantization method

`quark_mxfp4` online quantization method

Online NVFP4 to MXFP4 Requantization

Online FP8 to MXFP4 Requantization

Intel® Neural Compressor online quantization method

Diffusion Model Quantization on Ascend NPU

Online MXFP8

Offline MXFP8 (ModelSlim)

Reference