vllm.models.deepseek_v41.nvidia.specialized_decode.hopper.indexer_metadata
¶
The DeepSeek-V4.1 indexer metadata on SM90: the stock builder, with a step's prefill rows split over the TP ranks from less work.
Under tensor parallelism every rank holds the whole indexer: it scores each
prefill row of a step against the row's compressed keys and keeps the row's
top-k. The ranks can split the rows instead (../ampere/row_shard.py:
each rank scores a contiguous share, balanced by keys, and the ranks
exchange the int32 top-k rows).
HopperIndexerMetadataBuilder builds the stock metadata, then splits a
step's prefill rows once their scoring work (rows times compressed keys, an
upper bound) reaches ROW_SHARD_MIN_WORK: from there on, one exchange per
indexer layer costs less than the scoring it saves. On H100 at TP8 this
takes 2.3% off the time to first token of a 96K-token prompt. The SM90
indexer op (indexer.py) runs the stock op on its rank's rows and
exchanges them; it keeps the candidate rows of the two-level selection
replicated, and splitting those as well gained another 0.3% there.
Classes:
-
HopperIndexerBackend–The index K cache backend of the SM90 layer.
-
HopperIndexerMetadataBuilder–The stock indexer metadata, with prefill rows split over the TP ranks
Functions:
-
prefill_row_shard–The
tp_sizeranks' shares of a step's prefill rows (balanced by
HopperIndexerBackend
¶
Bases: DeepseekV41IndexerBackend
The index K cache backend of the SM90 layer.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/hopper/indexer_metadata.py
HopperIndexerMetadataBuilder
¶
Bases: DeepseekV32IndexerMetadataBuilder
The stock indexer metadata, with prefill rows split over the TP ranks
from ROW_SHARD_MIN_WORK.
Methods:
-
build–The stock metadata, whose prefill rows split when their scoring
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/hopper/indexer_metadata.py
build(common_prefix_len, common_attn_metadata, fast_build=False)
¶
The stock metadata, whose prefill rows split when their scoring work is large enough.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/hopper/indexer_metadata.py
prefill_row_shard(seq_lens_cpu, query_lens_cpu, compress_ratio, tp_size)
¶
The tp_size ranks' shares of a step's prefill rows (balanced by
keys), from the prefill requests' sequence lengths, their new tokens and
the index cache's compress_ratio; None below ROW_SHARD_MIN_WORK.