vllm.models.qwen3_5.nvidia.ops.fused_allreduce_norm_quant
¶
TP all-reduce + residual add + Gemma RMSNorm + consumer input quant.
Calls FlashInfer's kARResidualRMSNormOut{FP8,FP4}Quant patterns directly
(no torch.compile custom op). The kernel writes the bf16 normed activation
(GDN's in_proj_ba reads it) next to the static FP8 / NVFP4 codes; its
outputs are bitwise identical to FlashInfer AR+norm followed by
scaled_fp8_quant / scaled_fp4_quant. Dynamic per-token FP8 has no pattern
on vLLM's FlashInfer backends: it runs FlashInfer AR+norm, then
scaled_fp8_quant as the linear would.
Returns None when the fused kernel does not apply (FlashInfer or a supported workspace unavailable, or a batch above the workspace budget); the caller then runs an explicit all-reduce and the Triton norm-quant.
Functions:
-
fused_allreduce_gemma_rms_norm_quant–Fused
gemma_rms_norm(all_reduce(hidden_states) + residual)+ quant.
_get_workspace(hidden_states, tp_size)
¶
The FlashInfer quant workspace if it can take hidden_states, else
None.
Source code in vllm/models/qwen3_5/nvidia/ops/fused_allreduce_norm_quant.py
fused_allreduce_gemma_rms_norm_quant(hidden_states, residual, weight, eps, scale, key)
¶
Fused gemma_rms_norm(all_reduce(hidden_states) + residual) + quant.
hidden_states is the per-rank partial of a row-parallel linear;
scale is the consumer's static FP8 input scale or NVFP4 global scale
inverse, or None for dynamic per-token FP8. Returns the quantized normed
activation, the normed activation in hidden_states' dtype, and the new
residual (written into the hidden_states buffer), or None when the
fused kernel does not apply.
Source code in vllm/models/qwen3_5/nvidia/ops/fused_allreduce_norm_quant.py
99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 | |