vllm.models.mistral.nvidia.ops.indexer
¶
DSA indexer Q/K epilogues.
Q: SSMax, optional RoPE and per-head UE8M0 FP8 quantization in one launch,
with the quantization scale folded into the head weights (extends
fused_indexer_q_rope_quant from sparse_attn_indexer).
K: fp32 LayerNorm and optional RoPE in one launch, optionally quantized and
written straight to the paged FP8 indexer cache.
The Q launch can also clear the top-k buffer rows that the indexer fills.
The indexer layout is [rope | nope]. Intermediate roundings match the
unfused path.
Functions:
-
fused_indexer_k_norm_rope–Return
rope(LayerNorm_fp32(k))ink.dtype;kis[T, D]. -
fused_indexer_k_norm_rope_cache_write–Write
rope(LayerNorm_fp32(k))to the paged FP8 indexer cache, as -
fused_indexer_q_ssmax_rope_quant–Return FP8
q[T, H, D]and fp32weights * q_scale * scales.
fused_indexer_k_norm_rope(positions, k, weight, bias, eps, cos_sin_cache, is_neox)
¶
Return rope(LayerNorm_fp32(k)) in k.dtype; k is [T, D].
RoPE covers the leading cos_sin_cache.shape[1] dims (skipped if
cos_sin_cache is None).
Source code in vllm/models/mistral/nvidia/ops/indexer.py
fused_indexer_k_norm_rope_cache_write(positions, k, weight, bias, eps, cos_sin_cache, is_neox, kv_cache, slot_mapping)
¶
Write rope(LayerNorm_fp32(k)) to the paged FP8 indexer cache, as
indexer_k_quant_and_cache does with the bf16 result (UE8M0 scale per
128-dim row; tokens past slot_mapping or with a negative slot are
skipped).
Source code in vllm/models/mistral/nvidia/ops/indexer.py
fused_indexer_q_ssmax_rope_quant(positions, q, cos_sin_cache, weights, log_positions, s, b, softmax_scale, head_scale, is_neox, topk_indices=None)
¶
Return FP8 q [T, H, D] and fp32 weights * q_scale * scales.
SSMax is applied first, then RoPE on the leading cos_sin_cache.shape[1]
dims (skipped if cos_sin_cache is None), then one UE8M0 scale per head.
Rows [:T] of topk_indices, if given, are filled with -1.
Source code in vllm/models/mistral/nvidia/ops/indexer.py
167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 | |