Skip to content

vllm.models.muse_spark.amd.ops.fused_markov

The Muse Spark DSpark Markov rounds with one launch a round (--monokernel).

DSparkSpeculator._sample_sequential runs many small kernels in each of the 7 draft rounds: the Markov embedding of the previous token, the bias GEMV over the 202048-token draft vocabulary (207 MB of markov_w2), the add to the base logits, Muse's output transform (multiplier and soft cap), the copy into the d2t scatter buffer, the Gumbel sample (its kernel, the argmax over its blocks and a gather) and the copy into draft_tokens.

Here a round is one launch over the vocabulary in blocks of BLOCK tokens. A program gets the previous token of each row (the argmax over the previous round's block maxima), computes its block's bias for all rows with one read of markov_w2, and for each row adds the base logits, applies the transform, stores the round's logits into the draft logits cache and takes the Gumbel-max of its block with vLLM's own gumbel_noised_argmax. One small launch then writes the last round's tokens. The noise depends only on the request's seed, the round, the position and the token id, and both paths keep the first of equal maxima, so the sampled token does not depend on the block size.

The bias is an fp32 sum of 512 products rounded to bf16, as vLLM's GEMV output, but in another order. So a bias value can differ in its last bf16 bit. This changes only which tokens the draft proposes: the rejection sampler verifies every draft token against the target's distribution. The other steps are bitwise the same as vLLM's loop.

MuseSparkDSparkForCausalLM sets this module's sample_sequential as its own with --monokernel on ROCm. tests/kernels/models/amd/ test_muse_spark_fused_markov.py compares it with vLLM's loop.

Functions:

  • sample_sequential –

    DSparkSpeculator._sample_sequential with one launch a Markov round.

_scratch(device, vocab, use_fp64)

Two sets of block maxima and their token ids, one written by the current round and one read from the previous round. They are allocated once, so a CUDA graph keeps their addresses.

Source code in vllm/models/muse_spark/amd/ops/fused_markov.py
def _scratch(device, vocab, use_fp64):
    """Two sets of block maxima and their token ids, one written by the
    current round and one read from the previous round. They are allocated
    once, so a CUDA graph keeps their addresses."""
    key = (device, vocab, use_fp64)
    if key not in _buffers:
        nb = triton.cdiv(vocab, BLOCK)
        dt = torch.float64 if use_fp64 else torch.float32
        _buffers[key] = [
            (
                torch.empty(MAX_ROWS, nb, dtype=dt, device=device),
                torch.empty(MAX_ROWS, nb, dtype=torch.int64, device=device),
            )
            for _ in range(2)
        ]
    return _buffers[key]

_why_not(spec, num_reqs)

Why this step takes vLLM's loop, or None when the fused launches compute the same sampling. Every check is the same on all ranks.

Source code in vllm/models/muse_spark/amd/ops/fused_markov.py
def _why_not(spec, num_reqs: int) -> str | None:
    """Why this step takes vLLM's loop, or None when the fused launches
    compute the same sampling. Every check is the same on all ranks."""
    model = spec.model
    head = model.model.markov_head
    w1, w2 = head.markov_w1.weight, head.markov_w2.weight
    if not getattr(model, "d2t_is_identity", False):
        return "a d2t mapping that is not the identity"
    if spec.draft_watermarker is not None:
        return "a draft watermarker"
    if spec.use_confidence_head:
        return "the confidence head"
    if spec.acceptance_estimator is not None:
        return "the acceptance estimator"
    if not 1 <= num_reqs <= MAX_ROWS:
        return f"{num_reqs} requests"
    if w1.dtype != torch.bfloat16 or w2.dtype != torch.bfloat16:
        return "Markov weights not bf16"
    if w1.shape[1] != w2.shape[1] or not (w1.is_contiguous() and w2.is_contiguous()):
        return "Markov weight layout"
    if head.markov_w2.quant_method.__class__.__name__ != "UnquantizedEmbeddingMethod":
        return "a quantized Markov head"
    # The base logits are bf16 when the lm_head runs in bf16.
    bf16 = (None, torch.bfloat16)
    if (
        model.lm_head.weight.dtype not in bf16
        or model.logits_processor.head_dtype not in bf16
    ):
        return "base logits not bf16"
    cache = spec.draft_logits
    if cache is not None and (
        cache.row_mapping is not None
        or cache.values.dtype != torch.bfloat16
        or cache.values.shape[-1] != w2.shape[0]
    ):
        return "the draft logits cache layout"
    return None

sample_sequential(spec, num_reqs, head_hidden)

DSparkSpeculator._sample_sequential with one launch a Markov round. Returns False, having done nothing, when this step takes vLLM's loop.

Source code in vllm/models/muse_spark/amd/ops/fused_markov.py
def sample_sequential(spec, num_reqs: int, head_hidden: torch.Tensor) -> bool:
    """DSparkSpeculator._sample_sequential with one launch a Markov round.
    Returns False, having done nothing, when this step takes vLLM's loop."""
    why = _why_not(spec, num_reqs)
    if why is not None:
        logger.debug_once("Muse Spark fused Markov sampling off for a step: %s", why)
        return False
    model = spec.model
    n_spec = spec.num_speculative_steps
    num_sample = num_reqs * n_spec
    sample_hidden = head_hidden[spec.sample_indices[:num_sample]]
    base_logits = model.compute_draft_logits(sample_hidden)
    assert base_logits.dtype == torch.bfloat16 and base_logits.stride(-1) == 1
    vocab = base_logits.shape[-1]
    head = model.model.markov_head
    w1, w2 = head.markov_w1.weight, head.markov_w2.weight
    assert w2.shape[0] == vocab
    idx_map = spec.sample_idx_mapping[:num_sample].view(num_reqs, n_spec)
    sample_pos = spec.sample_pos[:num_sample].view(num_reqs, n_spec)
    has_cache = spec.draft_logits is not None
    cache = spec.draft_logits.values if has_cache else base_logits
    tokens = spec.draft_tokens
    use_fp64 = bool(spec.use_fp64_gumbel)
    soft_cap = model.output_soft_cap_temp
    bufs = _scratch(base_logits.device, vocab, use_fp64)
    nb = triton.cdiv(vocab, BLOCK)
    nb2 = triton.next_power_of_2(nb)
    rows_p2 = triton.next_power_of_2(num_reqs)
    logger.info_once(
        "Muse Spark fused Markov sampling: %d rounds, vocabulary %d, rank %d",
        n_spec, vocab, w1.shape[1],
    )  # fmt: skip
    for step in range(n_spec):
        prev_max, prev_arg = bufs[(step + 1) % 2]
        out_max, out_arg = bufs[step % 2]
        _markov_round_kernel[(nb,)](
            prev_max,
            prev_arg,
            spec.input_buffers.input_ids,
            spec._anchor_idx,
            tokens,
            tokens.stride(0),
            w1,
            w2,
            base_logits[step::n_spec],
            base_logits.stride(0) * n_spec,
            cache,
            cache.stride(0),
            cache.stride(1) if has_cache else 0,
            idx_map,
            idx_map.stride(0),
            sample_pos,
            sample_pos.stride(0),
            spec.temperature,
            spec.seeds,
            out_max,
            out_arg,
            step,
            num_reqs,
            vocab,
            nb,
            float(np.float32(model.output_multiplier)),
            float(soft_cap or 1.0),
            float(np.float32(1.0) / np.float32(soft_cap or 1.0)),
            RANK=w1.shape[1],
            BLOCK=BLOCK,
            K_CHUNK=K_CHUNK,
            ROWS_P2=rows_p2,
            DOT_N=max(16, rows_p2),
            BLOCKS_P2=nb2,
            HAS_CACHE=has_cache,
            SOFT_CAP=soft_cap is not None,
            USE_FP64=use_fp64,
            num_warps=NUM_WARPS,
        )
    last_max, last_arg = bufs[(n_spec - 1) % 2]
    _markov_last_kernel[(num_reqs,)](
        last_max,
        last_arg,
        tokens,
        tokens.stride(0),
        nb,
        STEP=n_spec - 1,
        BLOCKS_P2=nb2,
    )
    return True