Please confirm you are human
This browser or connection looks automated. Press and continuously hold the control for 3 seconds to enable Google-hosted web results and, when separately allowed, AI-assisted answers.
A successful check enables 100 search requests. Interactive access does not authorize scraping, systematic collection, or reuse of search output.
News
layernorm
2+ day, 33+ min ago (33+ words) vLLM _fused_add_rms_norm(x, x_residual, weight, epsilon, variance_size=None) ¶ Fused add and weighted root-mean-square layer normalization _rms_norm(x, weight, epsilon, variance_size=None) ¶ Weighted root-mean-square layer normalization...
triton_turboquant_unified_attention
3+ day, 14+ hour ago (180+ words) Unified (prefill + decode) Triton attention kernel for TurboQuant. Structure is ported from vllm/v1/attention/ops/triton_unified_attention.py (the AITER unified attention kernel already upstreamed into vLLM). Only the K and V load sites are replaced: instead of reading raw fp16 keys/values from…...
tq_decode
3+ day, 14+ hour ago (266+ words) vLLM Build a TQ decode kernel module. build_tq_decode_module(num_seqs, num_kv_heads, num_partitions, max_blocks_per_seq=512, softmax_scale=None, query_group_size=QUERY_GROUP_SIZE, kv_block_size=KV_BLOCK_SIZE, use_hw_v_transpose=False, tile_groups_per_partition=1) ¶ query_group_size (= num_query_heads // num_kv_heads) selects the GQA factor. Supported values: 8 and 16. For 8 the MFMA's 16-row capacity is half-used; lanes 8..15 compute garbage and are gated out of all global-memory writes via an OOB-offset trick on the…...
metrics
8+ mon, 3+ week ago (66+ words) Aggregate and log spec decoding metrics. Record spec decoding metrics in Prometheus. The acceptance rate can be calculated using a PromQL query: The mean acceptance length (conventionally including bonus tokens) can be calculated using: A per-position acceptance rate vector can…...
allreduce_rmsnorm_reduce_scatter_early_exit
2+ week, 3+ day ago (69+ words) vLLM Routed AllReduce/RMSNorm with CTA-specialized ReduceScatter early exit. One routed role plus one ReduceScatter role per destination group. Own and launch the routed AllReduce/RMSNorm plus shared ReduceScatter. Compile the rank/M specialization without retaining caller tensors. Validate constraints…...
fused_mla_key_concat_kv_cache
2+ week, 3+ day ago (277+ words) Fused MLA prefill and decode epilogues for Kimi-K3. The optional positions / cos_sin_cache pair enables GPT-J-style RoPE inside the epilogue. Omitting both keeps the K3 NoPE fast path. The kernels use Programmatic Dependent Launch to overlap the tail of the producing GEMMs on…...
chunk_intra_token_parallel
2+ week, 3+ day ago (40+ words) Token-parallel implementation: each token gets its own thread block. Writes directly to Aqk and Akk tensors (in-place). [B, T, H, K] [B, T, HV, BT] output tensor to write to [B, T, HV, BC] output tensor for diagonal blocks (fp32)...
kv_cache_manager
8+ mon, 3+ week ago (1126+ words) The allocation result of KVCacheManager, work as the interface between Scheduler and KVCacheManager, to hide KVCacheManager's internal data structure from the Scheduler. blocks[i][j] refers to the i-th kv_cache_group and the j-th block of tokens.We don't use block of…...
static_sink_attention
7+ mon, 5+ day ago (12+ words) vLLM Attention with static sink tokens Bases: Attention, CustomOp...
sharded_state_loader
9+ mon, 1+ week ago (82+ words) vLLM Model loader that directly loads each worker's model state dict, which enables a fast load path for large tensor-parallel models where each worker only needs to read its own shard rather than the entire checkpoint. See examples/features/sharded_state/save_sharded_state_offline.py…...