vllm.models.minimax_m3.amd.ops.nvfp4_sparse_decode.host
¶
Launch wrapper of the FlyDSL NVFP4 sparse decode.
Functions:
-
decode_splits–Splits of a row's blocks.
-
minimax_m3_nvfp4_sparse_decode_fly–Sparse decode attention over the NVFP4 K/V pages.
_grid_x(rows, ql, xcds, place)
¶
grid.x of the kernel's row placement (see kernel.py).
Source code in vllm/models/minimax_m3/amd/ops/nvfp4_sparse_decode/host.py
_query_xcd_count(device_index)
¶
The device's XCD count, or zero when HIP cannot report it.
Source code in vllm/models/minimax_m3/amd/ops/nvfp4_sparse_decode/host.py
_split_buffers(device)
cached
¶
Split partials and tickets; allocated in the warm-up, before capture.
Source code in vllm/models/minimax_m3/amd/ops/nvfp4_sparse_decode/host.py
_xcd_count(device_index)
cached
¶
8 on gfx950 (the topology the XCD grid was tuned on), else 0.
Source code in vllm/models/minimax_m3/amd/ops/nvfp4_sparse_decode/host.py
_xcd_plan(num_reqs, xcds, topk, decode_query_len, num_kv_heads, cus)
¶
(splits, row placement, waves_per_eu) of the XCD-aware grid, or None.
The kernel takes 259 VGPRs, so a CU runs one workgroup at a time, and a
second round of workgroups on an XCD costs a whole workgroup latency chain
(block-table, page and partial round trips: about 7 us cold) for a few
blocks saved. So the plan is the most splits (at most MAX_XCD_SPLITS)
for which the busiest XCD runs all its workgroups in one round:
PLACE_REQ (request r on XCD r % xcds) puts ql x ceil(reqs / xcds) rows
on the busiest XCD, PLACE_BAL ceil(rows / xcds) on every XCD;
PLACE_REQ on ties (a request's rows share an L2). With more rows than
an XCD has CUs, up to MAX_OCC2_REQS requests: 2 balanced splits built
for two workgroups a CU (waves_per_eu 2: 256 VGPRs, 2 spilled outside the
loop), whose overlapping latency chains beat a second round; past that,
None (the cost model on the plain grid).
Measured on MI355X (top-k 16, DSpark 7 = 8 rows a request, one KV head, 64K / 128K / 256K contexts, cold, graph replay): 1-8 requests 4 splits PLACE_REQ (unchanged); 9-10 requests 3 splits PLACE_BAL (14.5-15.7 us vs 16.2-17.1 with 2); 11-16 2 splits PLACE_REQ (unchanged); 17-32 no split, PLACE_REQ (22-24 us; 17-24 took 29-31 us with the former 4 splits, 25-32 already ran unsplit); 33-48 2 splits PLACE_BAL occupancy 2 (34.5-36.7 us vs 44-50 with the cost model's 4 splits). A request's KV heads also share its XCD, so with 2 or 4 KV heads per rank (TP2, TP1) the plan does not apply and the cost model is kept.
Source code in vllm/models/minimax_m3/amd/ops/nvfp4_sparse_decode/host.py
decode_splits(num_rows, num_blocks, cus)
¶
Splits of a row's blocks.
The kernel is VALU-bound, so the cost model is rounds of workgroups per CU times blocks per split plus about one block of fixed cost; the fewest splits among the minima, each split at least two blocks.
Source code in vllm/models/minimax_m3/amd/ops/nvfp4_sparse_decode/host.py
minimax_m3_nvfp4_sparse_decode_fly(q, kv_cache, topk_idx, block_table, seq_lens, sm_scale, output, k_scale=None, v_scale=None, decode_query_len=1, splits=None, xcd_count=None, out_q=None, out_scale=None, place=None, waves_per_eu=None)
¶
Sparse decode attention over the NVFP4 K/V pages.
Row i is query row i % decode_query_len of request i // decode_query_len
and attends to the tokens of its topk_idx blocks up to its position.
With out_q and out_scale the output is written as MXFP8 (E4M3 values
and row-major E8M0 scales per 32 dims, equal to mxfp8_e4m3_quantize of
the bf16 output) instead of bf16 into output, which is then unused.
splits / place / waves_per_eu default to the XCD plan (_xcd_plan),
else the cost model; an explicit splits keeps PLACE_REQ. For a given
split count the output does not depend on the placement or the occupancy
hint (the splits combine in split order).
Source code in vllm/models/minimax_m3/amd/ops/nvfp4_sparse_decode/host.py
180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 | |