vllm.model_executor.layers.quantization.input_hadamard
¶
Block Hadamard rotation of linear-layer inputs, independent of the weight format.
Rotating each contiguous block of the input dimension by an orthonormal
Hadamard matrix H on both operands leaves the product unchanged,
(x H)(w H)^T = x w^T, while spreading outliers across each block, which
shrinks the error of block-scaled quantization (MXFP4, MXFP8, ...).
Classes:
-
InputHadamardFusable–Mixin for a linear method that can take over the input rotation of
-
InputHadamardLinearMethod–Wraps a linear method and rotates the layer input by an orthonormal
Functions:
-
block_hadamard_rotate–Rotate each contiguous
hadamard.shape[0]-wide block of the last -
hadamard_matrix–Orthonormal Sylvester Hadamard matrix,
scipy.linalg.hadamard(size)
InputHadamardFusable
¶
Mixin for a linear method that can take over the input rotation of
InputHadamardLinearMethod, for example inside its activation quant.
Methods:
-
fuse_input_hadamard–Called after
process_weights_after_loading. Return True only if
Source code in vllm/model_executor/layers/quantization/input_hadamard.py
fuse_input_hadamard(layer, size)
¶
Called after process_weights_after_loading. Return True only if
apply will now rotate each contiguous size-wide block of the input
by the orthonormal Sylvester Hadamard matrix itself.
Source code in vllm/model_executor/layers/quantization/input_hadamard.py
InputHadamardLinearMethod
¶
Bases: LinearMethodBase
Wraps a linear method and rotates the layer input by an orthonormal
block Hadamard matrix before the wrapped apply, in the manner of
CompressedTensorsLinearTransformMethod.
With rotate_weight, the weight ([out, in]) is rotated the same way
right before the wrapped method processes it, for example before online
quantization. Otherwise the checkpoint stores the weight already rotated,
together with an int8 input_rotation tensor holding the unnormalized
(+-1) Sylvester matrix, which is checked and then dropped.
The activation rotation runs in PyTorch unless the wrapped method takes it
over through InputHadamardFusable.
Source code in vllm/model_executor/layers/quantization/input_hadamard.py
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 | |
_load_input_rotation(param, loaded_weight, *args, **kwargs)
¶
Every shard of a fused layer carries the same rotation, and it must
be the Sylvester matrix apply rotates by.
Source code in vllm/model_executor/layers/quantization/input_hadamard.py
block_hadamard_rotate(x, hadamard)
¶
Rotate each contiguous hadamard.shape[0]-wide block of the last
dimension of x by hadamard, computed in fp32 and returned in
x.dtype.
Source code in vllm/model_executor/layers/quantization/input_hadamard.py
hadamard_matrix(size, device)
cached
¶
Orthonormal Sylvester Hadamard matrix, scipy.linalg.hadamard(size)
scaled by 1 / sqrt(size), in fp32. size must be a power of two.