vllm.model_executor.warmup.kernel_warmup
¶
Warmup kernels used during model execution. This is useful specifically for JIT'ed kernels as we don't want JIT'ing to happen during model execution.
Functions:
-
flashinfer_autotune–Autotune FlashInfer operations.
_aiter_gemm_m_buckets(max_m)
¶
One token count per AITER tuned-GEMM bucket up to max_m.
AITER pads M to 1, 2, 4 or 8 below 16, then to multiples of 16 / 32 / 64 / 128 up to 256 / 1024 / 4096 / beyond, and builds each bucket's kernel on first use.
Source code in vllm/model_executor/warmup/kernel_warmup.py
_all_ranks_have_matching_cache(path, group)
¶
True iff every rank in group has its own, mutually consistent file.
Source code in vllm/model_executor/warmup/kernel_warmup.py
_autotune_cache_fingerprint(path)
¶
Identify a saved autotune file by its FlashInfer metadata and size.
Source code in vllm/model_executor/warmup/kernel_warmup.py
_flashinfer_deferred_moe_token_counts(runner)
¶
Return bounded token counts that exercise deferred MoE finalization.
Source code in vllm/model_executor/warmup/kernel_warmup.py
_load_shared_autotune_cache(cache_path, tuner, tune_group)
¶
Load the per-rank autotune cache into the tuner, group-wide.
Returns True only when every rank of the tune group had a matching file and loaded it: the dummy-run sweep contains collectives, so skipping it is safe only if the decision is unanimous.
Source code in vllm/model_executor/warmup/kernel_warmup.py
_outside_kv_cache_transient_window(worker)
¶
Keep a one-off tuning step out of the extensible KV cache's reserve.
An extensible KV cache keeps free the allocator peak its warmup steps reach
(reserved_headroom_bytes). Autotuning profiles every candidate tactic
over all token buckets on scratch buffers that serving never allocates
again, so its peak is not a serving transient. Record the transient reached
before tuning, then restart peak tracking afterwards; the warmup steps that
follow run with the tuned tactics and measure their own transients.
Source code in vllm/model_executor/warmup/kernel_warmup.py
flashinfer_autotune(runner, *, skip_attn=False)
¶
Autotune FlashInfer operations. FlashInfer have many implementations for the same operation, autotuning runs benchmarks for each implementation and stores the results. The results are cached transparently and future calls to FlashInfer will use the best implementation. Without autotuning, FlashInfer will rely on heuristics, which may be significantly slower.
Parameters:
-
(runner¶GPUModelRunner) –The model runner to tune.
-
(skip_attn¶bool, default:False) –Additionally skip attention in the tuning dummy runs.
With PP > 1, stages run different layers and may profile different ops, so each stage's TP group tunes separately with its own cache file; otherwise the world group tunes together. Per-tactic timings are averaged over the tuning group so all its ranks select the same tactic.
Legacy results are persisted per rank: FlashInfer keys MoE entries by tp/ep rank
(MoERunner.get_cache_key_extras), so one rank's file only hits on that
rank. A rank with a cache hit skips the per-tactic reduce the others block
in, so ranks keep loaded configs only if every rank in the tuning group
has a matching file and successfully loads it.
Source code in vllm/model_executor/warmup/kernel_warmup.py
600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 | |