Quantization#

AutoGPTQ#

Olive integrates AutoGPTQ for quantization.

AutoGPTQ is an easy-to-use LLM quantization package with user-friendly APIs, based on GPTQ algorithm (weight-only quantization). With GPTQ quantization, you can quantize your favorite language model to 8, 4, 3 or even 2 bits. This comes without a big drop of performance and with faster inference speed. This is supported by most GPU hardwares.

Olive consolidates the GPTQ quantization into a single pass called GptqQuantizer which supports tune GPTQ quantization with hyperparameters for trade-off between accuracy and speed.

Please refer to GptqQuantizer for more details about the pass and its config parameters.

Example Configuration#

{
    "type": "GptqQuantizer",
    "data_config": "wikitext2_train"
}

AutoAWQ#

AutoAWQ is an easy-to-use package for 4-bit quantized models and it speeds up models by 3x and reduces memory requirements by 3x compared to FP16. AutoAWQ implements the Activation-aware Weight Quantization (AWQ) algorithm for quantizing LLMs. AutoAWQ was created and improved upon from the original work from MIT.

Olive integrates AutoAWQ for quantization and make it possible to convert the AWQ quantized torch model to onnx model.

Please refer to AutoAWQQuantizer for more details about the pass and its config parameters.

Example Configuration#

{
    "type": "AutoAWQQuantizer",
    "bits": 4
}

QuaRot#

QuaRot is a technique that rotates the weights of a model to make them more conducive to quantization. It is based on the QuaRot paper but only performs offline weight rotation. Can be followed by a pass such as GPTQ to quantize the rotated model weights.

This pass only supports HuggingFace transformer PyTorch models.

Example Configuration#

{
    "type": "QuaRot",
    "rotate_mode": "hadamard"
}

SpinQuant#

SpinQuant is a technique simlar to QuaRot that rotates the weights of a model to make them more conducive to quantization. The rotation weights are trained on a calibration dataset to improve activation quantization quality. It is based on the SpinQuant paper but only performs offline weight rotation. Can be followed by a pass such as GPTQ to quantize the rotated model weights.

This pass only supports HuggingFace transformer PyTorch models.

Example Configuration#

{
    "type": "SpinQuant",
    "rotate_mode": "hadamard",
    "a_bits": 8
}

RTN#

RTN (Round To Nearest) is a fast, calibration-free weight quantization method that enables low-bit quantization of large models without relying on gradient-based optimization or calibration datasets. RTN quantization uses simple rounding to the nearest quantization level, making it extremely fast while maintaining reasonable accuracy.

This pass supports ONNX models and can quantize MatMul and Gather nodes to 4 or 8 bits with block-wise quantization.

Example Configuration#

{
    "type": "OnnxBlockWiseRtnQuantization"
}

PyTorch Native RTN#

The Rtn pass applies RTN weight quantization directly to a PyTorch (Hugging Face) model, before any ONNX export. Unlike OnnxBlockWiseRtnQuantization (which operates on an already-exported ONNX graph), Rtn runs on the HfModelHandler and replaces the weight storage of quantizable parameters in place with a quantized tensor representation, while keeping the surrounding modules (nn.Linear / nn.Embedding, and MoE experts) otherwise unchanged. This lets you compose it with other PyTorch quantization passes (see the Gptq example below) and share settings via per-module overrides.

By default Rtn quantizes nn.Linear weights and leaves the embeddings, the language-model head, and any Mixture-of-Experts (MoE) experts at full precision. Three independent category flags opt those groups in:

Flag	Default	Effect when `true`
`lm_head`	`false`	Also quantize the language-model head.
`embeds`	`false`	Also quantize the input embeddings.
`moe`	`false`	Also quantize MoE expert weights: classic per-expert `nn.ModuleList` experts (e.g. Mixtral, PhiMoE on older `transformers`) are always supported, and fused-expert modules are supported whenever the resolved experts class reports `is_transposed=False` (the K-last `(E, OUT, K)` layout used by most fused-experts architectures). Architectures that store a transposed `(E, K, OUT)` layout, such as gpt-oss, are rejected. Architectures whose experts class does not report `is_transposed` at all – either because it predates the `transformers` fused-experts refactor, or because it has not adopted it (e.g. llama4, aria) – are also rejected, since Olive cannot independently verify their layout.

The moe flag is fail-closed: when moe is false, every module under an experts subtree is skipped even if it looks like a plain nn.Linear. If the model config indicates an MoE architecture but Olive cannot resolve the experts subtree for that (unrecognized) architecture, the pass raises a clear error before modifying any parameter, rather than silently quantizing the experts.

Only weight parameters are quantized. Fused expert 2D bias parameters, when present, remain in full precision.

`moe` and ONNX export#

MoE quantization in Olive is storage-only: Olive does not export 3D fused-expert QuantTensors to ONNX. Attempting to torch.onnx.export a model with 3D-quantized experts raises a clear error directing you to Mobius / ORT GenAI ModelBuilder for the experts. Non-MoE parts (attention projections, router/gate, embeddings, lm_head) still export through the existing MatMulNBits / GatherBlockQuantized path.

`moe` and native PyTorch inference: force the `"eager"` experts implementation#

transformers lets a loaded MoE model pick its runtime forward strategy independently of the checkpoint’s on-disk layout, via model.set_experts_implementation(...) / config._experts_implementation ("eager", "grouped_mm", "batched_mm", …). Some non-"eager" strategies (e.g. "grouped_mm", which transformers may auto-select even on CPU) call weight.transpose(-2, -1) on the fused-experts weight before dispatching to their matmul kernel. Because Olive’s 3D fused-expert QuantTensor is storage-only (see above) and cannot represent a transpose without a lossy unpack/re-quantize round trip, this raises a RuntimeError at inference time – even for architectures whose checkpoint layout (is_transposed=False) is fully supported for quantization.

Workaround: after loading a moe=True-quantized checkpoint for native PyTorch inference (as opposed to consuming it via Mobius / ORT GenAI ModelBuilder), force the eager path once, before running any forward pass:

model.set_experts_implementation("eager")

This is tracked as a follow-up in #2619.

`modules_to_not_convert` and `overrides`#

modules_to_not_convert lists module-name patterns to exclude entirely, and overrides maps module-name patterns to per-module {"bits", "symmetric", "group_size"} settings. Both accept two key styles:

Plain strings keep the existing Hugging Face semantics (substring match for modules_to_not_convert, literal match for overrides).
re:-prefixed keys are treated as regular expressions matched with re.fullmatch (e.g. "re:model\\.layers\\.\\d+\\.mlp\\..*").

For safety, re: patterns are validated before use: overly long patterns and patterns with nested unbounded quantifiers (catastrophic-backtracking / ReDoS shapes such as (a+)+) are rejected with a clear error.

Resolution order and override precedence#

When multiple rules could apply to the same target, the first rule that matches wins, in this order:

modules_to_not_convert (hard exclude)
category flags (lm_head / embeds / moe) — hard excludes; overrides can never re-include what a category flag skipped
overrides
pass-level defaults (bits / group_size / sym)

When several overrides entries match the same target, precedence is insertion order in the config, first match wins — not “longest / most specific pattern”. Order your overrides from most specific to least specific accordingly.

Example Configuration#

{
    "type": "Rtn",
    "bits": 4,
    "group_size": 128,
    "sym": false,
    "embeds": true,
    "moe": true,
    "overrides": {
        "re:.*\\.experts\\..*": { "bits": 4, "group_size": 32 }
    }
}

Composing with `Gptq`#

Rtn can run on an already-quantized model, so you can quantize the transformer nn.Linear layers with a calibration-based pass such as Gptq first, then cover the parts Gptq doesn’t handle (embeddings, lm_head, MoE experts) with Rtn:

[
    { "type": "Gptq" },
    { "type": "Rtn", "moe": true, "embeds": true }
]

The reverse order is not supported: calibration-based passes assume a clean full-precision starting point and will reject an already-quantized model.

Migration note: removal of `QuantLinear` / `QuantEmbedding`#

The previous nn.Module wrappers olive.common.quant.nn.QuantLinear and QuantEmbedding have been removed as a sanctioned breaking change. Quantized weights are now stored as a QuantTensor on the parameter itself rather than by swapping the parent module. Checkpoints produced by the old QuantLinear / QuantEmbedding classes cannot be reloaded through Olive’s own HF quantizer — this includes both:

models persisted with torch.save(model) (pickling live QuantLinear / QuantEmbedding instances), and
safetensors/state-dict checkpoints, since the buffer naming convention changed from bare <module>.qweight / .scales / .qzeros to <module>.weight_qweight / weight_scales / weight_qzeros.

There is no migration shim for either case (consistent with every prior packing-format change to this module). Re-run the Rtn pass on the original full-precision model to regenerate a checkpoint in the current format.

2-bit quantization is not exportable to ONNX#

Rtn supports bits in {2, 4, 8} for the PyTorch quantized-checkpoint path, but the ONNX export-compat path (QuantLinearNbit) only supports 4-bit and 8-bit packing. Attempting to export a 2-bit QuantTensor to ONNX raises a clear ValueError at export time rather than silently producing an incorrect graph; 2-bit quantization remains usable for PyTorch-only workflows.

PyTorch Native KQuant#

The KQuant pass is a calibration-free weight quantizer that applies llama.cpp’s iterative weighted-least-squares k-quant search to PyTorch (Hugging Face) model weights. It supports the same lm_head, embeds, and moe category flags as Rtn; all default to false.

With moe=true, classic per-expert nn.ModuleList layouts are supported, as are fused expert weights whose experts module reports is_transposed=false (K-last (E, OUT, K)). Transposed (E, K, OUT) layouts and fused implementations with a missing or non-boolean is_transposed attribute are rejected rather than risking quantization along the wrong dimension. Only direct 3D expert weight parameters are quantized; expert biases remain in full precision.

{
    "type": "KQuant",
    "bits": 4,
    "group_size": 32,
    "moe": true
}

HQQ#

HQQ (Half-Quadratic Quantization) is a fast, calibration-free weight quantization method that enables low-bit quantization of large models without relying on gradient-based optimization. Unlike data-dependent approaches like GPTQ, HQQ uses half-quadratic splitting to minimize weight quantization error efficiently.

This pass only supports ONNX models, and will only quantize MatMul nodes to 4 bits.

Example Configuration#

{
    "type": "OnnxHqqQuantization"
}

AMD Quark#

Olive integrates AMD Quark, AMD’s deep learning model quantization toolkit for both PyTorch and ONNX models.

Olive consolidates Quark quantization into a single pass called QuarkQuantization that routes to the appropriate Quark backend based on the input model type:

ONNX models (ONNXModelHandler) are quantized through the quark.onnx API. This path supports static and dynamic quantization, a wide range of data types (Int8/UInt8, Int16/UInt16, BFP16, MX), and advanced algorithms such as CLE, SmoothQuant, GPTQ, AdaRound, AdaQuant, and BiasCorrection.
HuggingFace PyTorch models (HfModelHandler) are quantized through the quark.torch API for LLMs, supporting schemes such as uint4_wo_128, int4_wo_128, int8, fp8, and mxfp4, with AWQ/GPTQ/SmoothQuant/rotation algorithms and export to HF safetensors, ONNX, or GGUF formats.

QuarkQuantization requires amd-quark>=0.12.

Please refer to QuarkQuantization for more details about the pass and its config parameters.

Example Configuration#

a. Quantize an ONNX model (static quantization with calibration data)

{
    "type": "QuarkQuantization",
    "data_config": "calib_data_config",
    "global_config": {
        "activation": { "data_type": "UInt8", "calibration_method": "Percentile" },
        "weight": { "data_type": "Int8", "calibration_method": "MinMax" }
    }
}

b. Quantize a HuggingFace LLM (weight-only 4-bit with AWQ)

{
    "type": "QuarkQuantization",
    "quant_scheme": "uint4_wo_128",
    "quant_algo": "awq",
    "dataset": "pileval_for_awq_benchmark",
    "model_export": ["hf_format"]
}

Quantize with onnxruntime#

Quantization is a technique to compress deep learning models by reducing the precision of the model weights from 32 bits to 8 bits. This technique is used to reduce the memory footprint and improve the inference performance of the model. Quantization can be applied to the weights of the model, the activations of the model, or both.

There are two ways to quantize a model in onnxruntime:

Dynamic Quantization: Dynamic quantization calculates the quantization parameters (scale and zero point) for activations dynamically, which means there is no any requirement for the calibration dataset. These calculations increase the cost of inference, while usually achieve higher accuracy comparing to static ones.
Static Quantization: Static quantization method runs the model using a set of inputs called calibration data. In this way, user must provide a calibration dataset to calculate the quantization parameters (scale and zero point) for activations before quantizing the model.

Olive consolidates the dynamic and static quantization into a single pass called OnnxQuantization, and provide the user with the ability to tune both quantization methods and hyperparameter at the same time. If the user desires to only tune either of dynamic or static quantization, Olive also supports them through OnnxDynamicQuantization and OnnxStaticQuantization respectively.

Please refer to OnnxQuantization, OnnxDynamicQuantization and OnnxStaticQuantization for more details about the passes and their config parameters.

Note: If target execution provider is QNN EP, the model might need to be preprocessed before quantization. Please refer to QnnPreprocess for more details about the pass and its config parameters. This preprocessing step fuses operators unsupported by QNN EP and inserts necessary operators to make the model compatible with QNN EP.

Example Configuration#

a. Tune the parameters of the OlivePass with pre-defined searchable values

{
    "type": "OnnxQuantization",
    "data_config": "calib_data_config"
}

b. Select parameters to tune

{
    "type": "OnnxQuantization",
    // select per_channel to tune with "SEARCHABLE_VALUES".
    // other parameters will use the default value, not to be tuned.
    "per_channel": "SEARCHABLE_VALUES",
    "data_config": "calib_data_config"
}

c. Use default values of the OlivePass (no tuning in this way)

{
    "type": "OnnxQuantization",
    // set per_channel to "DEFAULT_VALUE"
    "per_channel": "DEFAULT_VALUE",
    "data_config": "calib_data_config"
}

d. Specify parameters with user defined values

"onnx_quantization": {
    "type": "OnnxQuantization",
    // set per_channel to True.
    "per_channel": true,
    "data_config": "calib_data_config"
}

Check out this file for an example implementation of "user_script.py" and "calib_data_config/dataloader_config/type".

Quantize with Intel® Neural Compressor#

In addition to the default onnxruntime quantization tool, Olive also integrates Intel® Neural Compressor.

Intel® Neural Compressor is a model compression tool across popular deep learning frameworks including TensorFlow, PyTorch, ONNX Runtime (ORT) and MXNet, which supports a variety of powerful model compression techniques, e.g., quantization, pruning, distillation, etc. As a user-experience-driven and hardware friendly tool, Intel® Neural Compressor focuses on providing users with an easy-to-use interface and strives to reach “quantize once, run everywhere” goal.

Olive consolidates the Intel® Neural Compressor dynamic and static quantization into a single pass called IncQuantization, and provide the user with the ability to tune both quantization methods and hyperparameter at the same time. If the user desires to only tune either of dynamic or static quantization, Olive also supports them through IncDynamicQuantization and IncStaticQuantization respectively.

Example Configuration#

"inc_quantization": {
    "type": "IncStaticQuantization",
    "approach": "weight_only",
    "weight_only_config": {
        "bits": 4,
        "algorithm": "gptq"
    },
    "data_config": "calib_data_config",
    "calibration_sampling_size": [8],
    "save_as_external_data": true,
    "all_tensors_to_one_file": true
}

Please refer to IncQuantization, IncDynamicQuantization and IncStaticQuantization for more details about the passes and their config parameters.

NVIDIA TensorRT Model Optimizer-Windows#

Olive also integrates TensorRT Model Optimizer-Windows

The TensorRT Model Optimizer-Windows is engineered to deliver advanced model compression techniques, including quantization, to Windows RTX PC systems. Specifically tailored to meet the needs of Windows users,it is optimized for rapid and efficient quantization, featuring local GPU calibration, reduced system and video memory consumption, and swift processing times.

The primary objective of the TensorRT Model Optimizer-Windows is to generate optimized, standards-compliant ONNX-format models for DirectML backends. This makes it an ideal solution for seamless integration with ONNX Runtime (ORT) and DirectML (DML) frameworks, ensuring broad compatibility with any inference framework supporting the ONNX standard.

Olive consolidates the NVIDIA TensorRT Model Optimizer-Windows quantization into a single pass called NVModelOptQuantization which supports AWQ and RTN algorithms.

Example Configuration#

a. Pure 4 bit Quantization#

"quantization": {
    "type": "NVModelOptQuantization",
    "algorithm": "awq",
    "tokenizer_dir": "microsoft/Phi-3-mini-4k-instruct",
    "calibration_method": "awq_lite"
}

b. Mixed Precision Quantization#

For better accuracy while maintaining model compression, you can enable mixed precision quantization using the enable_mixed_quant parameter. This allows higher precision levels (8 bit) for important layers of the model while using 4 bits for others:

1. Default Mixed Precision Strategy

"quantization": {
    "type": "NVModelOptQuantization",
    "algorithm": "awq",
    "tokenizer_dir": "meta-llama/Llama-3.1-8B-Instruct",
    "calibration_method": "awq_lite",
    "enable_mixed_quant": true
}

Configuration:

enable_mixed_quant is set to true, the default mixed precision strategy quantizes:
8 bit layer: Important layer are quantized 8-bits per-channel
4 bit layer: All other layers

2. Custom Layer Selection with layers_8bit

For fine-grained control over which specific layers should use INT8 quantization, use the layers_8bit parameter with a comma-separated list of layer name patterns:

"quantization": {
    "type": "NVModelOptQuantization",
    "algorithm": "awq",
    "tokenizer_dir": "meta-llama/Llama-3.1-8B-Instruct",
    "calibration_method": "awq_lite",
    "layers_8bit": "model.layers.0,model.layers.1,lm_head"
}

Configurations:

layers_8bit accepts comma-separated layer name patterns (e.g., "model.layers.0,lm_head")
Matching layers are quantized to 8 bit for better accuracy, they are quantized per-channel
Non-matching layers are quantized to 4 bit for efficiency
When layers_8bit is specified, mixed precision is automatically enabled
Overrides the default enable_mixed_quant strategy

Please refer to Phi3.5 example for usability and setup details.

Quantize with AI Model Efficiency Toolkit#

Olive supports quantizing models with Qualcomm’s AI Model Efficiency Toolkit (AIMET).

AIMET is a software toolkit for quantizing trained ML models to optimize deployment on edge devices such as mobile phones or laptops. AIMET employs post-training and fine-tuning techniques to minimize accuracy loss during quantization.

Olive consolidates AIMET quantization into a single pass called AimetQuantization which supports LPBQ, SeqMSE, and AdaRound. Multiple techniques can be applied in a single pass by listing them in the techniques array. If no techniques are specified, AIMET applies basic static quantization to the model using the provided data.

Technique	Description
LPBQ	An alternative to blockwise quantization which allows backends to leverage existing per-channel quantization kernels while significantly improving encoding granularity.
SeqMSE	Optimizes the weight encodings of each layer of a model to minimize the difference between the layer’s original and quantized outputs.
AdaRound	Tunes the rounding direction for quantized model weights to minimize the local quantization error at each layer output.

Example Configuration#

{
    "type": "AimetQuantization",
    "data_config": "calib_data_config"
}

LPBQ#

Configurations:

block_size: Number of input channels to group in each block (default: 64).
op_types: List of operator types for which to enable LPBQ (default: ["Gemm", "MatMul", "Conv"]).
nodes_to_exclude: List of node names to exclude from LPBQ weight quantization (default: None)

{
    "type": "AimetQuantization",
    "data_config": "calib_data_config",
    "techniques": [
        {"name": "lpbq", "block_size": 64}
    ]
}

SeqMSE#

Configurations:

data_config: Data config to use for SeqMSE optimization. Defaults to calibration set if not specified.
num_candidates: Number of encoding candidates to sweep for each weight (default: 20).

{
    "type": "AimetQuantization",
    "data_config": "calib_data_config",
    "precision": "int4",
    "techniques": [
        {"name": "seqmse", "num_candidates": 20}
    ]
}

AdaRound#

Configurations:

num_iterations: Number of optimization steps to take for each layer (default: 10000). Recommended value is 10K for weight bitwidths >= 8-bits, 15K for weight bitwidths < 8 bits.
nodes_to_exclude: List of node names to exclude from AdaRound optimization (default: None).

{
    "type": "AimetQuantization",
    "data_config": "calib_data_config",
    "techniques": [
        {"name": "adaround", "num_iterations": 10000, "nodes_to_exclude": ["/lm_head/MatMul"]}
    ]
}

Please refer to AimetQuantization for more details about the pass and its config parameters.

Quantization#

AutoGPTQ#

Example Configuration#

AutoAWQ#

Example Configuration#

QuaRot#

Example Configuration#

SpinQuant#

Example Configuration#

RTN#

Example Configuration#

PyTorch Native RTN#

moe and ONNX export#

moe and native PyTorch inference: force the "eager" experts implementation#

modules_to_not_convert and overrides#

Resolution order and override precedence#

Example Configuration#

Composing with Gptq#

Migration note: removal of QuantLinear / QuantEmbedding#

2-bit quantization is not exportable to ONNX#

PyTorch Native KQuant#

HQQ#

Example Configuration#

AMD Quark#

Example Configuration#

Quantize with onnxruntime#

Example Configuration#

Quantize with Intel® Neural Compressor#

Example Configuration#

NVIDIA TensorRT Model Optimizer-Windows#

Example Configuration#

a. Pure 4 bit Quantization#

b. Mixed Precision Quantization#

Quantize with AI Model Efficiency Toolkit#

Example Configuration#

LPBQ#

SeqMSE#

AdaRound#

`moe` and ONNX export#

`moe` and native PyTorch inference: force the `"eager"` experts implementation#

`modules_to_not_convert` and `overrides`#

Composing with `Gptq`#

Migration note: removal of `QuantLinear` / `QuantEmbedding`#