{ "hf_id": "moonshotai/Kimi-K3", "meta": { "title": "Kimi-K3", "slug": "kimi-k3", "provider": "Moonshot AI", "description": "Pre-release 2.8T-parameter native multimodal MoE with Kimi Delta Attention, Gated MLA, Attention Residuals, and a 1M-token context window", "date_added": "2026-07-27", "date_updated": "2026-10-26", "tasks": [ "multimodal", "text" ], "performance_headline": "Pre-release TP8, TEP16, TP8xPP2, and disaggregated P/D profiles for the 2.8T MXFP4 checkpoint", "related_recipes": [ "moonshotai/Kimi-K2.6", "moonshotai/Kimi-K2.5" ], "default_hardware": "b300", "hardware": { "h200": "verified", "b200": "verified", "b300": "verified", "gb200": "verified", "gb300": "verified", "mi355x": "verified", "ascend_910c": "verified" } }, "recommended_command": { "hardware": "b300", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": { "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_USE_RUST_FRONTEND": "1" }, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve moonshotai/Kimi-K3 \\\n --trust-remote-code \\\n --gpu-memory-utilization 0.95 \\\n --tensor-parallel-size 8 \\\n --load-format fastsafetensors \\\n --disable-flashinfer-autotune \\\n --max-model-len 1048576 \\\n --kv-cache-dtype fp8 \\\n --attention-config '{\"use_prefill_query_quantization\":true,\"mla_prefill_backend\":\"TOKENSPEED_MLA\"}' \\\n --enable-prefix-caching \\\n --attention-backend TOKENSPEED_MLA \\\n --prefix-match-unit 128 \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 \\\n --reasoning-parser kimi_k3", "argv": [ "vllm", "serve", "moonshotai/Kimi-K3", "--trust-remote-code", "--gpu-memory-utilization", "0.95", "--tensor-parallel-size", "8", "--load-format", "fastsafetensors", "--disable-flashinfer-autotune", "--max-model-len", "1048576", "--kv-cache-dtype", "fp8", "--attention-config", "{\"use_prefill_query_quantization\":true,\"mla_prefill_backend\":\"TOKENSPEED_MLA\"}", "--enable-prefix-caching", "--attention-backend", "TOKENSPEED_MLA", "--prefix-match-unit", "128", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3", "--reasoning-parser", "kimi_k3" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_ALLREDUCE_USE_FLASHINFER=1 \\\n -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \\\n -e VLLM_USE_V2_MODEL_RUNNER=1 \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:latest moonshotai/Kimi-K3 \\\n --trust-remote-code \\\n --gpu-memory-utilization 0.95 \\\n --tensor-parallel-size 8 \\\n --load-format fastsafetensors \\\n --disable-flashinfer-autotune \\\n --max-model-len 1048576 \\\n --kv-cache-dtype fp8 \\\n --attention-config '{\"use_prefill_query_quantization\":true,\"mla_prefill_backend\":\"TOKENSPEED_MLA\"}' \\\n --enable-prefix-caching \\\n --attention-backend TOKENSPEED_MLA \\\n --prefix-match-unit 128 \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 \\\n --reasoning-parser kimi_k3", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_ALLREDUCE_USE_FLASHINFER=1", "-e", "VLLM_ENGINE_READY_TIMEOUT_S=3600", "-e", "VLLM_USE_V2_MODEL_RUNNER=1", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai:latest", "moonshotai/Kimi-K3", "--trust-remote-code", "--gpu-memory-utilization", "0.95", "--tensor-parallel-size", "8", "--load-format", "fastsafetensors", "--disable-flashinfer-autotune", "--max-model-len", "1048576", "--kv-cache-dtype", "fp8", "--attention-config", "{\"use_prefill_query_quantization\":true,\"mla_prefill_backend\":\"TOKENSPEED_MLA\"}", "--enable-prefix-caching", "--attention-backend", "TOKENSPEED_MLA", "--prefix-match-unit", "128", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3", "--reasoning-parser", "kimi_k3" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "blackwell", "compute_arch": "sm103", "display_name": "B300", "description": "NVIDIA B300 SXM 268 GB HBM3e · 8-GPU HGX B300 node (Blackwell Ultra)", "gpu_count": 8, "vram_gb": 2144, "multi_node": false }, "alternatives": { "multi_node_tp": "/moonshotai/Kimi-K3/strategies/multi_node_tp.json", "multi_node_tep": "/moonshotai/Kimi-K3/strategies/multi_node_tep.json", "multi_node_tp_pp": "/moonshotai/Kimi-K3/strategies/multi_node_tp_pp.json", "multi_node_dep": "/moonshotai/Kimi-K3/strategies/multi_node_dep.json", "pd_cluster": "/moonshotai/Kimi-K3/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/moonshotai/Kimi-K3/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/moonshotai/Kimi-K3/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "b300": "/moonshotai/Kimi-K3/hw/b300.json", "h100": "/moonshotai/Kimi-K3/hw/h100.json", "h200": "/moonshotai/Kimi-K3/hw/h200.json", "b200": "/moonshotai/Kimi-K3/hw/b200.json", "gb200": "/moonshotai/Kimi-K3/hw/gb200.json", "gb300": "/moonshotai/Kimi-K3/hw/gb300.json", "mi300x": "/moonshotai/Kimi-K3/hw/mi300x.json", "mi325x": "/moonshotai/Kimi-K3/hw/mi325x.json", "mi355x": "/moonshotai/Kimi-K3/hw/mi355x.json" } }, "model": { "model_id": "moonshotai/Kimi-K3", "smg_grpc": { "reasoning_parser": "kimi_k3", "tool_call_parser": "kimi_k3" }, "min_vllm_version": "0.29.0", "default_frontend": "rust", "architecture": "moe", "parameter_count": "2.8T", "active_parameters": "16 experts/token + shared (of 896 routed)", "context_length": 1048576, "base_args": [ "--trust-remote-code", "--gpu-memory-utilization", "0.95" ], "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto", "note": "Use a K3-enabled vLLM nightly after the integration lands." }, "docker": { "command": "docker pull vllm/vllm-openai:latest", "note": "Use a K3-enabled nightly image after the integration lands. CUDA 13 (cu130) build only — no -cu129 tag; the host needs an r580+ NVIDIA driver." } } }, "features": { "tool_calling": { "description": "Enable automatic tool choice with the Kimi K3 tool-call parser.", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3" ] }, "reasoning": { "description": "Parse Kimi K3 reasoning output separately from the final answer.", "args": [ "--reasoning-parser", "kimi_k3" ] }, "spec_decoding": { "description": "Use DSpark speculative decoding.", "args": [ "--speculative-config", "{\"model\":\"RedHatAI/Kimi-K3-speculator.dspark\", \"num_speculative_tokens\":8, \"method\": \"dspark\", \"draft_sample_method\": \"probabilistic\", \"rejection_sample_method\": \"block\"}" ], "hardware_overrides": { "mi355x": { "args": [ "--kv-cache-dtype", "fp8", "--speculative-config", "{\"model\":\"Inferact/Kimi-K3-DSpark\",\"num_speculative_tokens\":7,\"method\":\"dspark\",\"attention_backend\":\"ROCM_AITER_MLA\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}" ] }, "amd": { "args": [ "--max-num-seqs", "128", "--speculative-config", "{\"model\":\"RedHatAI/Kimi-K3-speculator.dspark\", \"num_speculative_tokens\":8, \"method\": \"dspark\", \"draft_sample_method\": \"probabilistic\", \"rejection_sample_method\": \"block\"}" ] }, "gb200": { "args": [ "--speculative-config", "{\"model\":\"RedHatAI/Kimi-K3-speculator.dspark\", \"num_speculative_tokens\":8, \"method\": \"dspark\", \"attention_backend\":\"TOKENSPEED_MLA\", \"draft_sample_method\": \"probabilistic\", \"rejection_sample_method\": \"block\"}" ] }, "gb300": { "args": [ "--speculative-config", "{\"model\":\"RedHatAI/Kimi-K3-speculator.dspark\", \"num_speculative_tokens\":8, \"method\": \"dspark\", \"attention_backend\":\"TOKENSPEED_MLA\", \"draft_sample_method\": \"probabilistic\", \"rejection_sample_method\": \"block\"}" ] }, "npu": { "args": [ "--speculative-config", "{\"method\":\"dspark\",\"model\":\"RedHatAI/Kimi-K3-speculator.dspark\",\"num_speculative_tokens\":7,\"enforce_eager\":true}" ] } } }, "text_only": { "description": "Skip the vision encoder for text-only workloads. Mutually exclusive with encoder_parallel.", "args": [ "--language-model-only" ] } }, "opt_in_features": [ "spec_decoding", "text_only" ], "variants": { "default": { "precision": "mxfp4", "vram_minimum_gb": 1680, "supported_hardware": [ "h100", "h200", "b200", "b300", "gb200", "gb300", "mi300x", "mi325x", "mi355x" ], "description": "Quantization-aware-trained MXFP4 weights with MXFP8 activations; footprint is estimated until the checkpoint is published.", "hardware_overrides": { "mi355x": { "docker_image": "vllm/vllm-openai-rocm:nightly-rocm100" } } }, "nvfp4": { "precision": "nvfp4", "vram_minimum_gb": 1650, "supported_hardware": [ "b200", "b300", "gb200", "gb300" ], "model_id": "RedHatAI/Kimi-K3-NVFP4", "description": "RedHatAI compressed-tensors NVFP4 quantization; MoE layers quantized to NVFP4, attention and embeddings at full precision", "json": "/RedHatAI/Kimi-K3-NVFP4.json" }, "ascend_w4a8": { "model_id": "Eco-Tech/Kimi-K3-w4a8", "precision": "int4", "label": "W4A8 (Ascend)", "vram_minimum_gb": 1711, "supported_hardware": [ "ascend_910c" ], "extra_args": [ "--quantization", "ascend", "--enable-prefix-caching", "--enable-expert-parallel", "--max-num-seqs", "16", "--max-model-len", "131072", "--max-num-batched-tokens", "24576", "--gpu-memory-utilization", "0.9", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}", "--mm-processor-cache-gb", "0", "--additional-config", "{\"enable_cpu_binding\":true, \"enable_flashcomm1\":true}", "--mm-encoder-tp-mode", "data", "--limit-mm-per-prompt", "{\"vision_chunk\": 2}", "--allowed-local-media-path", "/" ], "extra_env": { "PYTORCH_NPU_ALLOC_CONF": "expandable_segments:True", "OMP_PROC_BIND": "false", "OMP_NUM_THREADS": "1", "TASK_QUEUE_ENABLE": "1", "HCCL_BUFFSIZE": "800", "VLLM_ENGINE_READY_TIMEOUT_S": "7200" }, "hardware_overrides": { "ascend_910c": { "docker_image": "quay.io/ascend/vllm-ascend:kimi-k3-a3" } }, "description": "ModelSlim INT4 W4A8 for Ascend NPU. Requires --quantization ascend. Validated on 4× 910C mixed P/D.", "json": "/Eco-Tech/Kimi-K3-w4a8.json" }, "fp8": { "model_id": "RedHatAI/Kimi-K3-FP8-BLOCK", "precision": "fp8", "vram_minimum_gb": 3360, "description": "RedHatAI FP8 block quantization", "json": "/RedHatAI/Kimi-K3-FP8-BLOCK.json" } }, "strategy_hardware": { "multi_node_tp_dp": { "h200": "supported", "gb200": "supported", "gb300": "supported", "ascend_910c": "supported" }, "multi_node_dep": { "h100": "unsupported", "h200": "unsupported", "ascend_910c": "unsupported" }, "multi_node_tp": { "ascend_910c": "unsupported" }, "multi_node_tep": { "ascend_910c": "unsupported" }, "multi_node_tp_pp": { "ascend_910c": "unsupported" }, "pd_cluster": { "ascend_910c": "unsupported" } }, "strategy_min_gpus": { "single_node_tp": 8, "multi_node_tp": 8, "multi_node_tp_dp": 16, "multi_node_tep": 8, "multi_node_tp_pp": 16, "multi_node_dep": 16, "h100": 32, "gb200": { "multi_node_tp": 16 }, "ascend_910c": 64 }, "hardware_overrides": { "hopper": { "extra_args": [ "--gpu-memory-utilization", "0.97", "--max-num-seqs", "5", "--max-model-len", "32768", "--moe-backend", "marlin", "--disable-custom-all-reduce", "--disable-flashinfer-autotune", "--max-num-batched-tokens", "4096", "--attention-backend", "FLASHMLA" ], "extra_env": { "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_USE_V2_MODEL_RUNNER": "1", "PYTORCH_CUDA_ALLOC_CONF": "expandable_segments:True" } }, "blackwell": { "extra_args": [ "--load-format", "fastsafetensors", "--disable-flashinfer-autotune", "--max-model-len", "1048576", "--kv-cache-dtype", "fp8", "--attention-config", "{\"use_prefill_query_quantization\":true,\"mla_prefill_backend\":\"TOKENSPEED_MLA\"}", "--enable-prefix-caching", "--attention-backend", "TOKENSPEED_MLA", "--prefix-match-unit", "128" ], "extra_env": { "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_USE_V2_MODEL_RUNNER": "1" } }, "amd": { "extra_env": { "VLLM_ROCM_USE_AITER": "1", "SAFETENSORS_FAST_GPU": "1", "VLLM_ROCM_USE_AITER_MOE_SITUV2": "1", "VLLM_USE_BREAKABLE_CUDAGRAPH": "0" }, "extra_args": [ "--load-format", "auto", "--gpu-memory-utilization", "0.95", "--mm-encoder-tp-mode", "data", "--max-num-seqs", "128", "--reasoning-parser", "kimi_k3", "--max-num-batched-tokens", "4096", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"custom_ops\":[\"+fused_rms_norm_gated\"]}" ] } }, "presets": { "gb200_agentx_agg": { "label": "AgentX aggregated", "hardware": "gb200", "strategy": "multi_node_tp_pp", "workload": "Agentic (InferenceX AgentX)", "default": true, "description": "TP8 x PP2 + DCP8 across four GB200 trays with the DSpark K4 draft and an embedded Mooncake store.", "source": { "label": "InferenceX #3748", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/3748", "key": "kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-agg", "revision": "b7eee97" }, "image": "vllm/vllm-openai:nightly-e68efc685ab1e6f1eda004ced7157e3ca411da2f", "select": { "frontend": "python", "features": { "spec_decoding": false } }, "args": [ "--tensor-parallel-size", "8", "--pipeline-parallel-size", "2", "--decode-context-parallel-size", "8", "--dcp-comm-backend", "a2a", "--attention-config", "{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true}", "--mamba-ssm-cache-dtype", "bfloat16", "--gpu-memory-utilization", "0.88", "--moe-backend", "auto", "--language-model-only", "--speculative-config", "{\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\"}" ], "env": { "VLLM_USE_DIRECT_DCP_A2A": "1", "VLLM_USE_DIRECT_DCP_Q_GATHER": "1", "VLLM_USE_DIRECT_DCP_KV_GATHER": "1", "VLLM_PREFIX_CACHE_RETENTION_INTERVAL": "0", "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4", "VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS": "0", "VLLM_RPC_TIMEOUT": "600000", "MC_SLICE_SIZE": "1048576", "MC_STORE_MEMCPY": "1", "MC_WORKERS_PER_CTX": "4", "MC_ENABLE_DEST_DEVICE_AFFINITY": "1", "UCX_MEMTYPE_CACHE": "n", "UCX_MEMTYPE_REG_WHOLE": "n" }, "json_args": { "--compilation-config": { "cudagraph_mode": "FULL_AND_PIECEWISE" } }, "points": { "c1": { "label": "1", "concurrency": 1, "args": [ "--max-num-seqs", "2" ], "json_args": { "--speculative-config": { "num_speculative_tokens": 7 }, "--compilation-config": { "cudagraph_capture_sizes": [ 8, 16, 64, 128, 256, 512, 1024, 2048, 4096, 8192 ] } } }, "c4": { "label": "4", "concurrency": 4, "args": [ "--max-num-seqs", "8" ], "json_args": { "--speculative-config": { "num_speculative_tokens": 7 }, "--compilation-config": { "cudagraph_capture_sizes": [ 8, 16, 24, 32, 40, 48, 56, 64, 128, 256, 512, 1024, 2048, 4096, 8192 ] } } }, "c8": { "label": "8", "concurrency": 8, "args": [ "--max-num-seqs", "16" ], "json_args": { "--speculative-config": { "num_speculative_tokens": 7 }, "--compilation-config": { "cudagraph_capture_sizes": [ 8, 16, 24, 40, 56, 64, 80, 104, 128, 256, 512, 1024, 2048, 4096, 8192 ] } } }, "c24": { "label": "24", "concurrency": 24, "args": [ "--max-num-seqs", "48" ], "json_args": { "--compilation-config": { "cudagraph_capture_sizes": [ 8, 16, 24, 40, 56, 64, 80, 104, 128, 256, 512, 1024, 2048, 4096, 8192 ] } } }, "c32": { "label": "32", "concurrency": 32, "default": true, "args": [ "--max-num-seqs", "64" ], "json_args": { "--compilation-config": { "cudagraph_capture_sizes": [ 1, 7, 18, 34, 53, 64, 75, 100, 128, 256, 512, 1024, 2048, 4096, 8192 ] } } }, "c48": { "label": "48", "concurrency": 48, "args": [ "--max-num-seqs", "96" ], "json_args": { "--compilation-config": { "cudagraph_capture_sizes": [ 1, 7, 18, 34, 53, 64, 75, 100, 128, 256, 512, 1024, 2048, 4096, 8192 ] } } }, "c56": { "label": "56", "concurrency": 56, "args": [ "--max-num-seqs", "112" ], "json_args": { "--compilation-config": { "cudagraph_capture_sizes": [ 1, 7, 18, 34, 53, 64, 75, 100, 128, 256, 512, 1024, 2048, 4096, 8192 ] } } } }, "runbook": "The GB200 AgentX lanes in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-*`, [SemiAnalysisAI/InferenceX#4322](https://github.com/SemiAnalysisAI/InferenceX/pull/4322), pinned to [`b7eee97`](https://github.com/SemiAnalysisAI/InferenceX/tree/b7eee972451f9297941d4cbe6d42d489d7ec8363/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx), which builds on #3748) serve the default `moonshotai/Kimi-K3` checkpoint (MXFP4 weights) with DSpark speculative decoding and a Mooncake KV store on 4-GPU NVL trays. They are benchmark configurations; the settings below are the production-useful parts. The builder's **Mooncake** option generates the `mooncake_master` and store config; the lane-specific connector settings are below. Throughput runs use InferenceX's golden acceptance lengths for DSpark.\n\nServed by vLLM's own frontend (no Dynamo) with an embedded RDMA Mooncake store (`global_segment_size` 200GB, `mlx5_0..mlx5_3`). The preset renders the TP8 x PP2 + DCP8 layout across four trays with the lane's K4/K7 draft, `--max-num-seqs` and CUDA graph sizes at each concurrency point. The builder's **Mooncake** KV Offload option renders a plain `MooncakeStoreConnector` with a 100GB segment; use the lane's connector and 200GB store from the command below.\n\n```bash\nexport VLLM_USE_DIRECT_DCP_A2A=1 VLLM_USE_DIRECT_DCP_Q_GATHER=1 VLLM_USE_DIRECT_DCP_KV_GATHER=1\nexport VLLM_ALLREDUCE_USE_FLASHINFER=1 NCCL_CUMEM_ENABLE=1 VLLM_USE_V2_MODEL_RUNNER=1\nexport VLLM_MOONCAKE_LOAD_RECV_THREADS=4 MC_SLICE_SIZE=1048576 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0\nexport MC_STORE_MEMCPY=1 MC_ENABLE_DEST_DEVICE_AFFINITY=1 MC_WORKERS_PER_CTX=4\nexport UCX_MEMTYPE_CACHE=n UCX_MEMTYPE_REG_WHOLE=n VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0\nexport VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 VLLM_RPC_TIMEOUT=600000\nvllm serve moonshotai/Kimi-K3 --trust-remote-code \\\n --tensor-parallel-size 8 --pipeline-parallel-size 2 \\\n --decode-context-parallel-size 8 --dcp-comm-backend a2a \\\n --attention-backend TOKENSPEED_MLA \\\n --attention-config '{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true}' \\\n --kv-cache-dtype fp8 --mamba-ssm-cache-dtype bfloat16 --enable-prefix-caching --prefix-match-unit 128 \\\n --max-model-len 1048576 --gpu-memory-utilization 0.88 --max-num-seqs --max-num-batched-tokens 8192 \\\n --speculative-config '{\"model\":\"Inferact/Kimi-K3-DSpark\",\"method\":\"dspark\",\"num_speculative_tokens\":,\"attention_backend\":\"TOKENSPEED_MLA\",\"draft_sample_method\":\"probabilistic\"}' \\\n --compilation-config '{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[,256,512,1024,2048,4096,8192]}' \\\n --kv-transfer-config '{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"enable_offload\":false}}' \\\n --load-format fastsafetensors --moe-backend auto --no-enable-flashinfer-autotune \\\n --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --enable-auto-tool-choice \\\n --language-model-only\n```\n", "json": "/moonshotai/Kimi-K3/presets/gb200_agentx_agg.json" }, "gb200_agentx_pd": { "label": "AgentX 1P1D", "hardware": "gb200", "strategy": "pd_cluster", "workload": "Agentic (InferenceX AgentX)", "default": true, "description": "One TP16 + DCP16 prefill pool and one decode pool across four GB200 trays each, served by Dynamo with the DSpark K4 draft.", "source": { "label": "InferenceX #4322", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4322", "key": "kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-disagg", "revision": "b7eee97" }, "image": { "vllm": "vllm/vllm-openai:nightly-e68efc685ab1e6f1eda004ced7157e3ca411da2f", "dynamo": "ffbbaaa6c44370e798687881317535ac17d66a76" }, "select": { "pd_router": "dynamo", "frontend": "rust", "features": { "tool_calling": false, "spec_decoding": false } }, "args": [ "--decode-context-parallel-size", "16", "--dcp-comm-backend", "a2a", "--attention-backend", "TOKENSPEED_MLA", "--attention-config", "{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true}", "--kv-cache-dtype", "fp8", "--mamba-ssm-cache-dtype", "bfloat16", "--enable-prefix-caching", "--prefix-match-unit", "128", "--enable-cumem-allocator", "--load-format", "safetensors", "--safetensors-load-strategy", "lazy", "--stream-interval", "10", "--language-model-only", "--speculative-config", "{\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\"}" ], "env": { "VLLM_USE_DIRECT_DCP_A2A": "1", "VLLM_USE_DIRECT_DCP_Q_GATHER": "1", "VLLM_USE_DIRECT_DCP_KV_GATHER": "1", "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_USE_NCCL_SYMM_MEM": "0", "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS": "0", "VLLM_PREFIX_CACHE_RETENTION_INTERVAL": "0", "NCCL_CUMEM_ENABLE": "1", "NCCL_MNNVL_ENABLE": "1", "NCCL_NVLS_ENABLE": "1", "NCCL_P2P_LEVEL": "NVL", "NCCL_NET_PLUGIN": "none", "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4", "MC_SLICE_SIZE": "1048576", "MC_ENABLE_DEST_DEVICE_AFFINITY": "1", "UCX_MEMTYPE_CACHE": "n", "UCX_MEMTYPE_REG_WHOLE": "n", "UCX_RCACHE_MAX_UNRELEASED": "1024", "UCX_TCP_AF_PRIO": "inet", "UCX_TLS": "all", "PYTORCH_CUDA_ALLOC_CONF": "max_split_size_mb:512", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800" }, "roles": { "prefill": { "parallelism": "tp", "nodes": 4, "args": [ "--enable-flashinfer-autotune", "--gpu-memory-utilization", "0.88", "--max-cudagraph-capture-size", "512" ], "env": { "VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT": "0" } }, "decode": { "parallelism": "tp", "nodes": 4, "args": [ "--no-data-parallel-hybrid-lb", "--max-num-batched-tokens", "1024", "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}" ], "env": { "VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT": "0" } } }, "points": { "c16": { "label": "16", "concurrency": 16, "default": true, "roles": { "prefill": { "args": [ "--max-num-seqs", "32" ] }, "decode": { "args": [ "--gpu-memory-utilization", "0.92", "--max-num-seqs", "32", "--max-cudagraph-capture-size", "160" ] } } }, "c32": { "label": "32", "concurrency": 32, "roles": { "prefill": { "args": [ "--max-num-seqs", "64" ] }, "decode": { "args": [ "--gpu-memory-utilization", "0.92", "--max-num-seqs", "64", "--max-cudagraph-capture-size", "320" ] } } }, "c48": { "label": "48", "concurrency": 48, "roles": { "prefill": { "args": [ "--max-num-seqs", "96" ] }, "decode": { "args": [ "--gpu-memory-utilization", "0.90", "--max-num-seqs", "96", "--max-cudagraph-capture-size", "480" ] } } } }, "runbook": "The GB200 AgentX lanes in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-*`, [SemiAnalysisAI/InferenceX#4322](https://github.com/SemiAnalysisAI/InferenceX/pull/4322), pinned to [`b7eee97`](https://github.com/SemiAnalysisAI/InferenceX/tree/b7eee972451f9297941d4cbe6d42d489d7ec8363/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/kimik3/vllm/gb200-fp4/agentx), which builds on #3748) serve the default `moonshotai/Kimi-K3` checkpoint (MXFP4 weights) with DSpark speculative decoding and a Mooncake KV store on 4-GPU NVL trays. They are benchmark configurations; the settings below are the production-useful parts. The builder's **Mooncake** option generates the `mooncake_master` and store config; the lane-specific connector settings are below. Throughput runs use InferenceX's golden acceptance lengths for DSpark.\n\nImage `vllm/vllm-openai:nightly-e68efc685ab1e6f1eda004ced7157e3ca411da2f` with Dynamo `ffbbaaa6c44370e798687881317535ac17d66a76`, etcd and an embedded RDMA Mooncake store. The lane's request plane is TCP (`DYN_REQUEST_PLANE=tcp`) and it starts no NATS service. Each role is one worker across four trays.\n\nRun each command on all four trays of the role with the same `--master-addr` (the first tray's address), `--nnodes 4`, `--node-rank 0` to `3`, and `--headless` on ranks 1-3. Prefill (`P_GPU`/`P_CAP` and `S` are the selected point's values):\n\n```bash\npython3 -m dynamo.vllm --model moonshotai/Kimi-K3 --served-model-name moonshotai/Kimi-K3 \\\n --disaggregation-mode prefill --request-plane tcp \\\n --nnodes 4 --node-rank <0-3> --master-addr --device-ids 0,1,2,3 \\\n --tensor-parallel-size 16 --decode-context-parallel-size 16 --dcp-comm-backend a2a \\\n --attention-backend TOKENSPEED_MLA \\\n --attention-config '{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true}' \\\n --speculative-config '{\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\"}' \\\n --kv-cache-dtype fp8 --mamba-ssm-cache-dtype bfloat16 --enable-prefix-caching --prefix-match-unit 128 --language-model-only \\\n --gpu-memory-utilization --max-num-seqs --max-cudagraph-capture-size \\\n --enable-cumem-allocator --stream-interval 10 --load-format safetensors --safetensors-load-strategy lazy \\\n --trust-remote-code \\\n --kv-events-config '{\"publisher\":\"zmq\",\"topic\":\"kv-events\",\"enable_kv_cache_events\":true,\"endpoint\":\"tcp://*:5200\"}' \\\n --kv-transfer-config '{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"enforce_handshake_compat\":false,\"enable_cross_layers_blocks\":false}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"compact_group_io\":true,\"max_load_batch_keys\":2,\"enable_cross_layers_blocks\":false,\"enable_offload\":false}}]}}'\n```\n\nDecode (`D_GPU`/`D_CAP` and `S` are the selected point's values). `--max-num-batched-tokens 1024` bounds the FlashInfer MoE autotune to the token counts decode can reach (at most 480); at vLLM's 16384-token default the autotune bucket needs a 2.9 GiB workspace that runs out of memory at 0.92, and the other ranks hang:\n\n```bash\npython3 -m dynamo.vllm --model moonshotai/Kimi-K3 --served-model-name moonshotai/Kimi-K3 \\\n --disaggregation-mode decode --request-plane tcp \\\n --nnodes 4 --node-rank <0-3> --master-addr --device-ids 0,1,2,3 \\\n --tensor-parallel-size 16 --decode-context-parallel-size 16 --dcp-comm-backend a2a \\\n --attention-backend TOKENSPEED_MLA \\\n --attention-config '{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true}' \\\n --speculative-config '{\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\"}' \\\n --kv-cache-dtype fp8 --mamba-ssm-cache-dtype bfloat16 --enable-prefix-caching --prefix-match-unit 128 --language-model-only \\\n --gpu-memory-utilization --max-num-seqs --max-num-batched-tokens 1024 --max-cudagraph-capture-size \\\n --enable-cumem-allocator --stream-interval 10 --load-format safetensors --safetensors-load-strategy lazy \\\n --trust-remote-code \\\n --kv-transfer-config '{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"enforce_handshake_compat\":false,\"enable_cross_layers_blocks\":false}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"compact_group_io\":true,\"max_load_batch_keys\":2,\"enable_cross_layers_blocks\":false,\"enable_offload\":false}}]}}'\n```\n\nFor tool calling and reasoning output, add `--reasoning-parser kimi_k3 --dyn-tool-call-parser kimi_k3 --dyn-reasoning-parser kimi_k3 --dyn-enable-structural-tag` to both roles. `python3 -m dynamo.vllm` parses the remaining arguments as vLLM engine arguments only, so `--enable-auto-tool-choice` and `--tool-call-parser` are rejected there.\n\n- **Environment (both roles):** the three direct-DCP variables, `VLLM_ALLREDUCE_USE_FLASHINFER=1`, `VLLM_USE_V2_MODEL_RUNNER=1`, `VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT=0`, `VLLM_USE_NCCL_SYMM_MEM=0`, `VLLM_SSM_CONV_STATE_LAYOUT=DS`, `VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0`, `NCCL_MNNVL_ENABLE=1`, `NCCL_NVLS_ENABLE=1`, `NCCL_CUMEM_ENABLE=1`, `NCCL_P2P_LEVEL=NVL`, `NCCL_NET_PLUGIN=none`, `VLLM_MOONCAKE_LOAD_RECV_THREADS=4`, `MC_SLICE_SIZE=1048576`, `MC_ENABLE_DEST_DEVICE_AFFINITY=1`, `UCX_MEMTYPE_CACHE=n`, `UCX_MEMTYPE_REG_WHOLE=n`, `UCX_RCACHE_MAX_UNRELEASED=1024`, `UCX_TCP_AF_PRIO=inet`, `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0`, `VLLM_ENGINE_READY_TIMEOUT_S=3600`, `VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800`, `ETCD_LEASE_TTL=600`, `PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512` and `DYN_REQUEST_PLANE=tcp`.\n- **Frontend:** the Dynamo frontend runs with `--router-mode least-loaded --router-session-affinity-ttl-secs 900`, so each conversation sticks to one worker and reuses its prefix cache. The Rust frontend is selected with `VLLM_USE_RUST_FRONTEND=1`, with `DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600` and `DYN_TOKENIZER_CACHE_BYTES=8589934592`. AgentX sends the conversation ID in the `X-Dynamo-Session-Id` request header.\n- **Mooncake master:** `--default_kv_lease_ttl=60000 --eviction_high_watermark_ratio=0.95 --eviction_ratio=0.10`, 128GB global segment, 4GB local buffer, RDMA over `mlx5_0..mlx5_3`.\n", "json": "/moonshotai/Kimi-K3/presets/gb200_agentx_pd.json" }, "gb300_agentx_pd": { "label": "AgentX xPyD", "hardware": "gb300", "strategy": "pd_cluster", "workload": "Agentic (InferenceX AgentX)", "default": true, "description": "TP8 + DCP8 prefill and decode workers across two GB300 trays each, served by Dynamo with the DSpark K4 draft.", "source": { "label": "InferenceX #4047", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4047", "key": "kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg" }, "image": { "vllm": "vllm/vllm-openai:nightly-e4a5b5537344b103ed10de7764378ef09604ef3d", "dynamo": "a0de541ea8c861d5ac01d15953ca529bea68637b" }, "select": { "pd_router": "dynamo", "frontend": "rust", "features": { "tool_calling": false, "spec_decoding": false } }, "args": [ "--decode-context-parallel-size", "8", "--dcp-comm-backend", "a2a", "--attention-backend", "TOKENSPEED_MLA", "--kv-cache-dtype", "fp8", "--mamba-ssm-cache-dtype", "float32", "--cp-kv-cache-interleave-size", "1", "--enable-prefix-caching", "--prefix-match-unit", "128", "--gpu-memory-utilization", "0.95", "--max-cudagraph-capture-size", "512", "--enable-cumem-allocator", "--no-enable-flashinfer-autotune", "--load-format", "fastsafetensors", "--stream-interval", "10", "--language-model-only", "--speculative-config", "{\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}" ], "env": { "VLLM_USE_DIRECT_DCP_A2A": "1", "VLLM_USE_DIRECT_DCP_Q_GATHER": "1", "VLLM_USE_DIRECT_DCP_KV_GATHER": "1", "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_USE_NCCL_SYMM_MEM": "0", "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_PREFIX_CACHE_RETENTION_INTERVAL": "0", "NCCL_CUMEM_ENABLE": "1", "NCCL_MNNVL_ENABLE": "1", "NCCL_NVLS_ENABLE": "1", "NCCL_P2P_LEVEL": "NVL", "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4", "MC_SLICE_SIZE": "1048576", "MC_MAX_MR_SIZE": "4294967296", "MC_ENABLE_DEST_DEVICE_AFFINITY": "1", "UCX_MEMTYPE_CACHE": "n", "UCX_MEMTYPE_REG_WHOLE": "n", "UCX_RCACHE_MAX_UNRELEASED": "1024", "UCX_TCP_AF_PRIO": "inet", "TOKENSPEED_MLA_FP16_PARTIALS": "0", "UCX_TLS": "all", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800", "VLLM_RPC_TIMEOUT": "600000" }, "roles": { "prefill": { "nodes": 2, "args": [ "--moe-backend", "deep_gemm_mega_moe", "--attention-config", "{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"trtllm_ragged_backend\":\"cute-dsl\"}", "--max-num-seqs", "32", "--max-cudagraph-capture-size", "8192", "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[4,8,12,16,20,24,32,40,48,64,80,96,128,160,256,384,512,640,768,896,1024,1152,1280,1408,1536,1664,1792,1920,2048,2304,2560,2816,3072,3328,3584,3840,4096,4608,5120,5632,6144,6656,7168,7680,8192]}" ], "env": { "VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT": "0", "VLLM_ENABLE_GEMM_RS": "1", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT": "1024", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT": "1024" } }, "decode": { "parallelism": "tp", "nodes": 2, "args": [ "--no-data-parallel-hybrid-lb", "--max-num-batched-tokens", "16384", "--additional-config", "{\"decode_backend\":\"latency_specialized\"}", "--max-num-seqs", "12", "--max-cudagraph-capture-size", "64", "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64]}", "--attention-config", "{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true}" ], "env": { "VLLM_ENABLE_GEMM_RS": "0" } } }, "runbook": "The GB300 AgentX disaggregated lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg`, [SemiAnalysisAI/InferenceX#4047](https://github.com/SemiAnalysisAI/InferenceX/pull/4047), points refined in [#4373](https://github.com/SemiAnalysisAI/InferenceX/pull/4373) and [#4516](https://github.com/SemiAnalysisAI/InferenceX/pull/4516)) runs Dynamo + vLLM with several prefill and decode workers. Prefill and decode workers are both TP8 + DCP8 across two 4-GPU GB300 trays; prefill workers also use EP8 with MegaMoE, and decode workers use no expert parallelism. KV moves over NIXL, and all workers share an RDMA Mooncake store. Image `vllm/vllm-openai:nightly-e4a5b5537344b103ed10de7764378ef09604ef3d`, Dynamo `a0de541ea8c861d5ac01d15953ca529bea68637b`.\n\n| Clients | Prefill workers | Decode workers | GPUs | Decode profile | Decode `--max-num-seqs` |\n|---:|---:|---:|---:|---|---:|\n| 16 | 1 | 3 | 32 | [Specialized](#specialized-decode-low-concurrency) | 12 |\n| 24-32 | 1 | 2 | 24 | [Specialized](#specialized-decode-low-concurrency) | 12 |\n| 48-64 | 1 | 2 | 24 | Regular | 128 |\n| 240 | 2 | 3 | 40 | Regular | 128 |\n| 240 | 3 | 4 | 56 | Regular | 128 |\n\nPrefill workers always use `--max-num-seqs 32`. The regular profile is the Workers commands below; the specialized one changes both roles as described under [Specialized decode](#specialized-decode-low-concurrency).\n\nThe preset renders one prefill pool and one decode pool of the specialized profile (the 1P2D c24-c32 row); its decode uses `--max-num-seqs 12`. For the regular points use the Workers commands below. It turns **Spec decoding** off, because the toggle's generic `RedHatAI/Kimi-K3-speculator.dspark` K8 draft was not qualified and specialized decode allows at most 7 draft tokens, and **Tool calling** off (see the parser note under Workers). This preset's DCP8 pools do not combine with a DEP decode pool, which renders TP1 with `--decode-context-parallel-size 8` that vLLM rejects; the DEP16 decode lane is the [AgentX 3P1D DEP16 preset](#preset-gb300_agentx_pd_dep16).\n\nDocumented here only: worker counts beyond one pool per role, the Mooncake store connector (the builder's Mooncake option renders a plain `MooncakeStoreConnector` without the qualified NIXL MultiConnector, recompute policy and load options), the Mooncake services, prefill `--kv-events-config`, and the frontend's router mode and session affinity (the builder renders round-robin).\n\n### Services\n\nStart etcd, NATS and a Mooncake master once:\n\n```bash\netcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://:2379\nprintf 'max_payload: 33554432\\n' > nats.conf && nats-server -c nats.conf\nmooncake_master --enable_http_metadata_server=true \\\n --default_kv_lease_ttl=60000 --eviction_high_watermark_ratio=0.95 --eviction_ratio=0.10\n```\n\nThe workers' Mooncake store uses P2P handshake metadata, RDMA over the node's NICs, a 160GB global segment and a 4GB local buffer, with offload disabled.\n\n### Environment (every worker)\n\n```bash\nexport VLLM_USE_DIRECT_DCP_A2A=1 VLLM_USE_DIRECT_DCP_Q_GATHER=1 VLLM_USE_DIRECT_DCP_KV_GATHER=1\nexport VLLM_ALLREDUCE_USE_FLASHINFER=1 VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT=0 VLLM_USE_NCCL_SYMM_MEM=0\nexport NCCL_CUMEM_ENABLE=1 NCCL_MNNVL_ENABLE=1 NCCL_NVLS_ENABLE=1 NCCL_P2P_LEVEL=NVL\nexport VLLM_USE_V2_MODEL_RUNNER=1 VLLM_USE_RUST_FRONTEND=1 VLLM_SSM_CONV_STATE_LAYOUT=DS\nexport VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0\nexport VLLM_MOONCAKE_LOAD_RECV_THREADS=4 MC_SLICE_SIZE=1048576 MC_MAX_MR_SIZE=4294967296 MC_ENABLE_DEST_DEVICE_AFFINITY=1\nexport UCX_MEMTYPE_CACHE=n UCX_MEMTYPE_REG_WHOLE=n UCX_RCACHE_MAX_UNRELEASED=1024 UCX_TCP_AF_PRIO=inet\nexport TOKENSPEED_MLA_FP16_PARTIALS=0\nexport DYN_REQUEST_PLANE=tcp DYN_TCP_REQUEST_TIMEOUT=120 ETCD_LEASE_TTL=600\nexport VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 VLLM_RPC_TIMEOUT=600000\n```\n\nSet `VLLM_ENABLE_GEMM_RS=1` on prefill workers and `VLLM_ENABLE_GEMM_RS=0` on decode workers. Prefill workers also set `DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT=1024` and `DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT=1024`. Loading the 2.8T checkpoint across trays and autotuning takes a while, which is why the engine-ready, execute-model and RPC timeouts are raised.\n\n### Workers\n\nEach worker spans two trays. Run the command on both trays with the same `--master-addr` (the first tray's address), `--nnodes 2` and `--node-rank 0` or `1`.\n\nPrefill:\n\n```bash\npython3 -m dynamo.vllm --model moonshotai/Kimi-K3 --served-model-name moonshotai/Kimi-K3 \\\n --disaggregation-mode prefill --request-plane tcp \\\n --nnodes 2 --node-rank <0|1> --master-addr --device-ids 0,1,2,3 \\\n --tensor-parallel-size 8 --decode-context-parallel-size 8 --dcp-comm-backend a2a \\\n --enable-expert-parallel --moe-backend deep_gemm_mega_moe \\\n --attention-backend TOKENSPEED_MLA \\\n --attention-config '{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"trtllm_ragged_backend\":\"cute-dsl\"}' \\\n --speculative-config '{\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}' \\\n --kv-cache-dtype fp8 --mamba-ssm-cache-dtype bfloat16 \\\n --enable-prefix-caching --prefix-match-unit 128 \\\n --gpu-memory-utilization 0.95 --max-num-seqs 32 --max-cudagraph-capture-size 8192 \\\n --compilation-config '{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[4,8,12,16,20,24,32,40,48,64,80,96,128,160,256,384,512,640,768,896,1024,1152,1280,1408,1536,1664,1792,1920,2048,2304,2560,2816,3072,3328,3584,3840,4096,4608,5120,5632,6144,6656,7168,7680,8192]}' \\\n --enable-cumem-allocator --enable-flashinfer-autotune \\\n --load-format fastsafetensors --stream-interval 10 --language-model-only --trust-remote-code \\\n --kv-events-config '{\"publisher\":\"zmq\",\"topic\":\"kv-events\",\"enable_kv_cache_events\":true,\"endpoint\":\"tcp://*:5200\"}' \\\n --kv-transfer-config '{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"enforce_handshake_compat\":false,\"enable_cross_layers_blocks\":false}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"enable_cross_layers_blocks\":false,\"enable_offload\":false}}]}}'\n```\n\nDecode (regular profile, `--max-num-seqs 128`):\n\n```bash\npython3 -m dynamo.vllm --model moonshotai/Kimi-K3 --served-model-name moonshotai/Kimi-K3 \\\n --disaggregation-mode decode --request-plane tcp \\\n --nnodes 2 --node-rank <0|1> --master-addr --device-ids 0,1,2,3 \\\n --tensor-parallel-size 8 --decode-context-parallel-size 8 --dcp-comm-backend a2a \\\n --attention-backend TOKENSPEED_MLA \\\n --attention-config '{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"tokenspeed_mla_min_split_kv\":8}' \\\n --speculative-config '{\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}' \\\n --kv-cache-dtype fp8 --mamba-ssm-cache-dtype bfloat16 \\\n --enable-prefix-caching --prefix-match-unit 128 \\\n --gpu-memory-utilization 0.95 --max-num-seqs 128 --max-cudagraph-capture-size 512 --use-replayssm \\\n --enable-cumem-allocator --enable-flashinfer-autotune \\\n --load-format fastsafetensors --stream-interval 10 --language-model-only --trust-remote-code \\\n --kv-transfer-config '{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"enforce_handshake_compat\":false,\"enable_cross_layers_blocks\":false}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"enable_cross_layers_blocks\":false,\"enable_offload\":false}}]}}'\n```\n\nFor tool calling and reasoning output, add `--reasoning-parser kimi_k3 --dyn-tool-call-parser kimi_k3 --dyn-reasoning-parser kimi_k3 --dyn-enable-structural-tag` to both roles. `python3 -m dynamo.vllm` parses the remaining arguments as vLLM engine arguments only, so the vLLM API-server flags `--enable-auto-tool-choice` and `--tool-call-parser` are rejected there.\n\n### Specialized decode (low concurrency)\n\nThe c16-c32 points use the `latency_specialized` decode backend from [GB300 low-latency preset](#preset-gb300_low_latency) on the decode workers (1P3D for c16, 1P2D for c24-c32), with the Workers commands above changed as follows. The backend's requirements are listed there: V2 model runner (set in Environment), FULL decode CUDA graphs, BF16 convolution state with FP32 recurrent state, DSpark with at most 7 draft tokens, no pipeline or expert parallelism, and DCP of 1 or TP. The decode workers already meet the last two.\n\nBoth roles:\n\n```bash\n--mamba-ssm-cache-dtype float32 --cp-kv-cache-interleave-size 1 --no-enable-flashinfer-autotune\n# replacing --mamba-ssm-cache-dtype bfloat16 and --enable-flashinfer-autotune\n```\n\nPin `--cp-kv-cache-interleave-size 1` on both roles. Left unset, NIXL P/D raises the DCP KV interleave to the KV block size after the specialized decode has read it, so its spec-verify causal mask uses the stale value.\n\nDecode only:\n\n```bash\n--additional-config '{\"decode_backend\":\"latency_specialized\"}' \\\n--attention-config '{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true}' \\\n--max-num-seqs 12 --max-cudagraph-capture-size 64 \\\n--compilation-config \"{\\\"cudagraph_mode\\\":\\\"FULL_AND_PIECEWISE\\\",\\\"cudagraph_capture_sizes\\\":[$(seq -s, 1 64)]}\"\n# replacing the regular decode's --attention-config (it also sets tokenspeed_mla_min_split_kv),\n# --max-num-seqs 128 and --max-cudagraph-capture-size 512, and without --use-replayssm\n```\n\n`--max-num-seqs` is `64 // (K + 1)` with the K4 draft, so a full verify batch (12 x 5 = 60 tokens) fits the 64-token graph, and CUDA graphs are captured for every batch size up to 64. The prefill workers do not set `--additional-config`: the backend rejects the expert parallelism they use. The uncapped c48+ points keep the regular profile, so TTFT is not traded for ITL by a standing decode queue.\n\n### Frontend\n\n```bash\nexport DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600 DYN_TOKENIZER_CACHE_BYTES=8589934592 DYN_TCP_REQUEST_TIMEOUT=120\nexport DYN_DISABLE_OPENAI_STRICT_SCHEMA_VALIDATION=1\npython3 -m dynamo.frontend --router-mode least-loaded --router-session-affinity-ttl-secs 1\n```\n\n`DYN_DISABLE_OPENAI_STRICT_SCHEMA_VALIDATION=1` skips the frontend's preflight validation of tool parameter schemas marked `strict: true`. AgentX sends the conversation ID in the `X-Dynamo-Session-Id` request header.\n", "json": "/moonshotai/Kimi-K3/presets/gb300_agentx_pd.json" }, "gb300_agentx_pd_dep16": { "label": "AgentX 3P1D DEP16", "hardware": "gb300", "strategy": "pd_cluster", "workload": "Agentic (InferenceX AgentX)", "description": "Three TP8 + EP8 prefill workers across two GB300 trays each and one DEP16 decode worker across four trays, served by Dynamo with the DSpark K4 draft and online MXFP8 target projections.", "source": { "label": "InferenceX #4516", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4516", "key": "kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dep16-disagg" }, "image": { "vllm": "vllm/vllm-openai:nightly-e4a5b5537344b103ed10de7764378ef09604ef3d", "dynamo": "a0de541ea8c861d5ac01d15953ca529bea68637b" }, "select": { "pd_router": "dynamo", "frontend": "rust", "features": { "tool_calling": false, "spec_decoding": false } }, "args": [ "--attention-backend", "TOKENSPEED_MLA", "--kv-cache-dtype", "fp8", "--mamba-ssm-cache-dtype", "bfloat16", "--prefix-match-unit", "128", "--gpu-memory-utilization", "0.95", "--enable-expert-parallel", "--enable-ep-weight-filter", "--moe-backend", "deep_gemm_mega_moe", "--max-num-seqs", "32", "--enable-cumem-allocator", "--enable-flashinfer-autotune", "--load-format", "fastsafetensors", "--stream-interval", "10", "--language-model-only", "--speculative-config", "{\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}" ], "remove_args": [ "--disable-flashinfer-autotune" ], "env": { "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_ENABLE_GEMM_RS": "1", "VLLM_USE_NCCL_SYMM_MEM": "0", "VLLM_USE_V2_MODEL_RUNNER": "1", "NCCL_CUMEM_ENABLE": "1", "NCCL_MNNVL_ENABLE": "1", "NCCL_NVLS_ENABLE": "1", "NCCL_P2P_LEVEL": "NVL", "NVSHMEM_REMOTE_TRANSPORT": "none", "NVSHMEM_DISABLE_MNNVL": "0", "NVSHMEM_IB_ENABLE_IBGDA": "0", "NVSHMEM_ENABLE_NIC_PE_MAPPING": "1", "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4", "MC_SLICE_SIZE": "1048576", "MC_MAX_MR_SIZE": "4294967296", "MC_ENABLE_DEST_DEVICE_AFFINITY": "1", "UCX_MEMTYPE_CACHE": "n", "UCX_MEMTYPE_REG_WHOLE": "n", "UCX_RCACHE_MAX_UNRELEASED": "1024", "UCX_TCP_AF_PRIO": "inet", "UCX_TLS": "all", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800", "VLLM_RPC_TIMEOUT": "600000" }, "roles": { "prefill": { "parallelism": "tep", "nodes": 2, "args": [ "--quantization-config", "{\"targets\":{\"*.in_proj_qkvgfab\":\"mxfp8\",\"*.fused_qkv_a_g_proj\":\"mxfp8\",\"*.routed_expert_down_proj\":\"mxfp8\",\"*.routed_expert_up_proj\":\"mxfp8\"}}", "--enable-prefix-caching", "--max-num-batched-tokens", "16384", "--max-cudagraph-capture-size", "8192", "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[4,8,12,16,20,24,32,40,48,64,80,96,128,160,256,384,512,640,768,896,1024,1152,1280,1408,1536,1664,1792,1920,2048,2304,2560,2816,3072,3328,3584,3840,4096,4608,5120,5632,6144,6656,7168,7680,8192]}", "--attention-config", "{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"trtllm_ragged_backend\":\"cute-dsl\"}" ], "env": { "VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT": "0", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT": "1024", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT": "1024" } }, "decode": { "parallelism": "dep", "nodes": 4, "args": [ "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}", "--quantization-config", "{\"targets\":{\"*.in_proj_qkvgfab\":\"mxfp8\",\"*.fused_qkv_a_g_proj\":\"mxfp8\",\"*.routed_expert_down_proj\":\"mxfp8\",\"*.routed_expert_up_proj\":\"mxfp8\",\"*.o_proj\":\"mxfp8\",\"*.shared_experts.gate_up_proj\":\"mxfp8\",\"*.shared_experts.down_proj\":\"mxfp8\"}}", "--max-num-batched-tokens", "512", "--max-cudagraph-capture-size", "512", "--use-replayssm", "--attention-config", "{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"tokenspeed_mla_min_split_kv\":8}" ] } }, "runbook": "The GB300 AgentX 3P1D lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dep16-disagg`, [SemiAnalysisAI/InferenceX#4516](https://github.com/SemiAnalysisAI/InferenceX/pull/4516)) runs Dynamo + vLLM with three prefill workers and one decode worker at 336 clients (40 GPUs). Each prefill worker is TP8 + EP8 across two 4-GPU GB300 trays, without DCP. The decode worker is DEP16 (`--tensor-parallel-size 1 --data-parallel-size 16` with expert parallelism) across four trays. Both roles use MegaMoE, `--max-num-seqs 32` (per data-parallel rank on decode) and online MXFP8 for selected target projections. KV moves over NIXL, and all workers share an RDMA Mooncake store. Image `vllm/vllm-openai:nightly-e4a5b5537344b103ed10de7764378ef09604ef3d`, Dynamo `a0de541ea8c861d5ac01d15953ca529bea68637b`.\n\nThe preset renders one prefill pool and the decode pool. It turns **Spec decoding** off, because the toggle's generic `RedHatAI/Kimi-K3-speculator.dspark` K8 draft was not qualified, and **Tool calling** off (see the parser note under Workers).\n\nDocumented here only: the three prefill workers, the Mooncake store connector (the builder renders a plain NIXL connector, without the qualified MultiConnector, recompute policy and load options), the Mooncake services, prefill `--kv-events-config`, `NVSHMEM_HCA_LIST`, and the frontend's router mode and session affinity (the builder renders round-robin).\n\n**Target quantization:** `--quantization-config` quantizes these target layers online to MXFP8: the KDA `in_proj_qkvgfab` and MLA `fused_qkv_a_g_proj` projections and the latent-MoE `routed_expert_up_proj`/`routed_expert_down_proj` projections on both roles, plus `o_proj` and the shared experts (`gate_up_proj`, `down_proj`) on decode. The DSpark draft is not quantized: vLLM builds the named draft from its own unquantized config, and the lane's logs show the online quantization pass only for the target layers.\n\n### Services\n\nStart etcd, NATS and a Mooncake master once, as for the [AgentX xPyD preset](#preset-gb300_agentx_pd): `mooncake_master --enable_http_metadata_server=true --default_kv_lease_ttl=60000 --eviction_high_watermark_ratio=0.95 --eviction_ratio=0.10`. The workers' Mooncake store uses P2P handshake metadata, RDMA over the node's NICs, a 160GB global segment and a 4GB local buffer, with offload disabled.\n\n### Environment (every worker)\n\n```bash\nexport VLLM_ALLREDUCE_USE_FLASHINFER=1 VLLM_ENABLE_GEMM_RS=1 VLLM_USE_NCCL_SYMM_MEM=0\nexport NCCL_CUMEM_ENABLE=1 NCCL_MNNVL_ENABLE=1 NCCL_NVLS_ENABLE=1 NCCL_P2P_LEVEL=NVL\nexport NVSHMEM_REMOTE_TRANSPORT=none NVSHMEM_DISABLE_MNNVL=0 NVSHMEM_IB_ENABLE_IBGDA=0 NVSHMEM_ENABLE_NIC_PE_MAPPING=1\nexport NVSHMEM_HCA_LIST=\nexport VLLM_USE_V2_MODEL_RUNNER=1 VLLM_USE_RUST_FRONTEND=1 VLLM_SSM_CONV_STATE_LAYOUT=DS\nexport VLLM_MOONCAKE_LOAD_RECV_THREADS=4 MC_SLICE_SIZE=1048576 MC_MAX_MR_SIZE=4294967296 MC_ENABLE_DEST_DEVICE_AFFINITY=1\nexport UCX_MEMTYPE_CACHE=n UCX_MEMTYPE_REG_WHOLE=n UCX_RCACHE_MAX_UNRELEASED=1024 UCX_TCP_AF_PRIO=inet\nexport DYN_REQUEST_PLANE=tcp DYN_TCP_REQUEST_TIMEOUT=120 ETCD_LEASE_TTL=600\nexport VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 VLLM_RPC_TIMEOUT=600000\n```\n\nPrefill workers also set `VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT=0`, `DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT=1024` and `DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT=1024`.\n\n### Workers\n\nPrefill (three workers; run each on its two trays with the same `--master-addr`, the first tray's address, `--nnodes 2` and `--node-rank 0` or `1`):\n\n```bash\npython3 -m dynamo.vllm --model moonshotai/Kimi-K3 --served-model-name moonshotai/Kimi-K3 \\\n --disaggregation-mode prefill --request-plane tcp \\\n --nnodes 2 --node-rank <0|1> --master-addr --device-ids 0,1,2,3 \\\n --tensor-parallel-size 8 --enable-expert-parallel --enable-ep-weight-filter --moe-backend deep_gemm_mega_moe \\\n --quantization-config '{\"targets\":{\"*.in_proj_qkvgfab\":\"mxfp8\",\"*.fused_qkv_a_g_proj\":\"mxfp8\",\"*.routed_expert_down_proj\":\"mxfp8\",\"*.routed_expert_up_proj\":\"mxfp8\"}}' \\\n --attention-backend TOKENSPEED_MLA \\\n --attention-config '{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"trtllm_ragged_backend\":\"cute-dsl\"}' \\\n --speculative-config '{\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}' \\\n --kv-cache-dtype fp8 --mamba-ssm-cache-dtype bfloat16 \\\n --enable-prefix-caching --prefix-match-unit 128 \\\n --gpu-memory-utilization 0.95 --max-num-seqs 32 --max-num-batched-tokens 16384 --max-cudagraph-capture-size 8192 \\\n --compilation-config '{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[4,8,12,16,20,24,32,40,48,64,80,96,128,160,256,384,512,640,768,896,1024,1152,1280,1408,1536,1664,1792,1920,2048,2304,2560,2816,3072,3328,3584,3840,4096,4608,5120,5632,6144,6656,7168,7680,8192]}' \\\n --enable-cumem-allocator --enable-flashinfer-autotune \\\n --load-format fastsafetensors --stream-interval 10 --language-model-only --trust-remote-code \\\n --kv-events-config '{\"publisher\":\"zmq\",\"topic\":\"kv-events\",\"enable_kv_cache_events\":true,\"endpoint\":\"tcp://*:5200\"}' \\\n --kv-transfer-config '{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"enforce_handshake_compat\":false,\"enable_cross_layers_blocks\":false}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"enable_cross_layers_blocks\":false,\"enable_offload\":false}}]}}'\n```\n\nDecode (one DEP16 worker; run on each of its four trays with `--data-parallel-start-rank` 0, 4, 8 and 12 and the same `--data-parallel-address`, the first tray's address):\n\n```bash\npython3 -m dynamo.vllm --model moonshotai/Kimi-K3 --served-model-name moonshotai/Kimi-K3 \\\n --disaggregation-mode decode --request-plane tcp --device-ids 0,1,2,3 \\\n --tensor-parallel-size 1 --data-parallel-size 16 --data-parallel-size-local 4 \\\n --data-parallel-start-rank <0|4|8|12> --data-parallel-address --data-parallel-rpc-port \\\n --data-parallel-hybrid-lb --enable-expert-parallel --enable-ep-weight-filter --moe-backend deep_gemm_mega_moe \\\n --quantization-config '{\"targets\":{\"*.in_proj_qkvgfab\":\"mxfp8\",\"*.fused_qkv_a_g_proj\":\"mxfp8\",\"*.routed_expert_down_proj\":\"mxfp8\",\"*.routed_expert_up_proj\":\"mxfp8\",\"*.o_proj\":\"mxfp8\",\"*.shared_experts.gate_up_proj\":\"mxfp8\",\"*.shared_experts.down_proj\":\"mxfp8\"}}' \\\n --attention-backend TOKENSPEED_MLA \\\n --attention-config '{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"tokenspeed_mla_min_split_kv\":8}' \\\n --speculative-config '{\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}' \\\n --kv-cache-dtype fp8 --mamba-ssm-cache-dtype bfloat16 \\\n --no-enable-prefix-caching --prefix-match-unit 128 \\\n --gpu-memory-utilization 0.95 --max-num-seqs 32 --max-num-batched-tokens 512 --max-cudagraph-capture-size 512 --use-replayssm \\\n --enable-cumem-allocator --enable-flashinfer-autotune \\\n --load-format fastsafetensors --stream-interval 10 --language-model-only --trust-remote-code \\\n --kv-transfer-config '{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"enforce_handshake_compat\":false,\"enable_cross_layers_blocks\":false}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_consumer\",\"kv_load_failure_policy\":\"recompute\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"enable_lookup\":false,\"enable_cross_layers_blocks\":false,\"enable_offload\":false}}]}}'\n```\n\nFor tool calling and reasoning output, add `--reasoning-parser kimi_k3 --dyn-tool-call-parser kimi_k3 --dyn-reasoning-parser kimi_k3 --dyn-enable-structural-tag` to both roles; `python3 -m dynamo.vllm` rejects the vLLM API-server flags `--enable-auto-tool-choice` and `--tool-call-parser`.\n\n### Frontend\n\n```bash\nexport DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600 DYN_TOKENIZER_CACHE_BYTES=8589934592 DYN_TCP_REQUEST_TIMEOUT=120\nexport DYN_DISABLE_OPENAI_STRICT_SCHEMA_VALIDATION=1\npython3 -m dynamo.frontend --router-mode least-loaded --router-session-affinity-ttl-secs 1\n```\n", "json": "/moonshotai/Kimi-K3/presets/gb300_agentx_pd_dep16.json" }, "gb300_low_latency": { "label": "Low latency", "hardware": "gb300", "strategy": "multi_node_tp", "workload": "Low latency (client concurrency 1-4)", "description": "One aggregated engine on the latency_specialized decode backend with the DSpark draft: TP16/DCP16 over four trays at concurrency 1, TP8/DCP8 over two at 2-4.", "source": { "label": "recipes #1134", "href": "https://github.com/vllm-project/recipes/pull/1134" }, "image": "vllm/vllm-openai:nightly", "select": { "frontend": "python", "features": { "spec_decoding": false } }, "env": { "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION": "1", "VLLM_USE_DIRECT_DCP_A2A": "1", "VLLM_USE_DIRECT_DCP_Q_GATHER": "1", "VLLM_USE_DIRECT_DCP_KV_GATHER": "1", "VLLM_PREFIX_CACHE_RETENTION_INTERVAL": "0", "VLLM_RPC_TIMEOUT": "600000", "VLLM_LOG_STATS_INTERVAL": "1" }, "args": [ "--dcp-comm-backend", "a2a", "--additional-config", "{\"decode_backend\":\"latency_specialized\"}", "--attention-backend", "TOKENSPEED_MLA", "--attention-config", "{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true}", "--kv-cache-dtype", "fp8", "--mamba-ssm-cache-dtype", "float32", "--gpu-memory-utilization", "0.92", "--max-num-seqs", "8", "--max-num-batched-tokens", "16384", "--speculative-config", "{\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--enable-prefix-caching", "--prefix-match-unit", "128", "--disable-flashinfer-autotune", "--moe-backend", "auto", "--load-format", "fastsafetensors", "--stream-interval", "10", "--language-model-only", "--trust-remote-code" ], "points": { "tp16": { "label": "1 (TP16, 4 trays)", "concurrency": 1, "default": true, "select": { "nodes": 4 }, "args": [ "--decode-context-parallel-size", "16", "--max-cudagraph-capture-size", "48" ], "json_args": { "--speculative-config": { "num_speculative_tokens": 5 }, "--compilation-config": { "cudagraph_mode": "FULL_AND_PIECEWISE", "cudagraph_capture_sizes": [ 5, 6, 10, 12, 20, 24, 40, 48 ] } } }, "tp8": { "label": "2–4 (TP8, 2 trays)", "concurrency": 4, "select": { "nodes": 2 }, "args": [ "--decode-context-parallel-size", "8", "--max-cudagraph-capture-size", "64" ], "json_args": { "--compilation-config": { "cudagraph_mode": "FULL_AND_PIECEWISE", "cudagraph_capture_sizes": [ 7, 8, 14, 16, 28, 32, 56, 64 ] } } } }, "runbook": "The `latency_specialized` decode backend provides a CuTe-based implementation for low-concurrency Kimi-K3 serving on Blackwell. The configurations below serve the default [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) MXFP4 checkpoint with [Inferact/Kimi-K3-DSpark](https://huggingface.co/Inferact/Kimi-K3-DSpark), using one aggregated engine across 4-GPU GB300 NVLink compute nodes and the native Python vLLM frontend.\n\nThe explicit opt-in is `--additional-config '{\"decode_backend\":\"latency_specialized\"}'`; `auto` keeps the stock decode path in this image. The specialized path requires the V2 model runner, FULL decode CUDA graphs, and BF16 convolution state with FP32 recurrent state (`--mamba-ssm-cache-dtype float32`). Keep `FULL_AND_PIECEWISE` and the capture sizes together: the lists cover DSpark draft and verification shapes. Prefill and eager/piecewise steps still use the stock implementation. This recipe uses PP1, no expert parallelism, and at most seven draft tokens; do not combine it with `--enforce-eager`, ReplaySSM/RecoverSSM, or adaptive verification. See the pinned [backend requirements](https://github.com/vllm-project/vllm/blob/49cb2bbf71cfd36ee5aa33d1e184b1cddff55294/vllm/models/kimi_k3/nvidia/specialized_decode/runtime/__init__.py).\n\nMake the same target checkpoint available at the same local path on every node, and make `Inferact/Kimi-K3-DSpark` available through the mounted Hugging Face cache. Run the preset's commands inside the nightly image on every allocated node, with four visible GPUs, host networking, sufficient shared memory, and the devices and network access required by your NVLink/RDMA deployment. Use the reachable address of rank 0 for `MASTER_ADDR` on every node and a distinct `NODE_RANK` from `0` through `NODES - 1`. The lane served the checkpoint from a local path with `--served-model-name moonshotai/Kimi-K3` and `--device-ids 0,1,2,3`; the builder serves the Hugging Face id.\n", "json": "/moonshotai/Kimi-K3/presets/gb300_low_latency.json" }, "mi355x_agentx": { "label": "AgentX TP8", "hardware": "mi355x", "strategy": "single_node_tp", "workload": "Agentic (InferenceX AgentX)", "default": true, "description": "The lane's every-concurrency settings on one MI355X node, for any concurrency; the per-concurrency cells are the AgentX cells preset.", "source": { "label": "InferenceX #4393", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4393", "key": "kimik3-fp4-mi355x-vllm-agentic-mtp" }, "args": [ "--gpu-memory-utilization", "0.9", "--max-model-len", "1048576", "--enable-prefix-caching", "--kv-cache-dtype", "fp8", "--attention-config", "{\"mla_prefill_backend\":\"ROCM_AITER_FA\"}", "--stream-interval", "10" ], "env": { "VLLM_ROCM_AITER_MLA_ASM_PADDING": "asm", "VLLM_ROCM_AITER_MLA_DCP_VERIFY": "asm", "AITER_FLYDSL_STAGE2_FP8": "1" }, "runbook": "The single-node MI355X TP8 AgentX configuration follows [InferenceX #4393](https://github.com/SemiAnalysisAI/InferenceX/pull/4393): its [master entry](https://github.com/SemiAnalysisAI/InferenceX/blob/dee7824e8ca3810cc36fe668ed75a927183af41f/inferencex-e2e/configs/amd-master.yaml#L766) and [lane recipe](https://github.com/SemiAnalysisAI/InferenceX/blob/dee7824e8ca3810cc36fe668ed75a927183af41f/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3/vllm/mi355x-fp4-mtp/agentic.yaml) at `dee7824`. It serves text with tool-call and reasoning parsers, using DCP8/A2A at every point. There is no concurrency-2 point.\n\n**In the command builder.** Settings the lane uses at every concurrency are part of the MI355X TP8 command: `--gpu-memory-utilization 0.9`, `--max-model-len 1048576`, `--enable-prefix-caching`, `--kv-cache-dtype fp8`, `--attention-config '{\"mla_prefill_backend\":\"ROCM_AITER_FA\"}'`, `--stream-interval 10`, `VLLM_ROCM_AITER_MLA_ASM_PADDING=asm`, `VLLM_ROCM_AITER_MLA_DCP_VERIFY=asm` and `AITER_FLYDSL_STAGE2_FP8=1`. Weights load with vLLM's default loader (the AMD profile's `--load-format auto`): with `fastsafetensors`, the lane's servers ran out of GPU memory while loading weights on its image. The lane's `--moe-backend auto` is vLLM's default selection, so the builder leaves that flag unset. Other AMD GPUs keep the general AMD profile.\n\n- **Spec decoding** serves the lane's draft, the published [`Inferact/Kimi-K3-DSpark`](https://huggingface.co/Inferact/Kimi-K3-DSpark), with its attention on `ROCM_AITER_MLA` ([vllm-project/vllm#55966](https://github.com/vllm-project/vllm/pull/55966)), FP8 KV, probabilistic drafting and block verification. Draft length is tuned per cell in the matrix below.\n- **Target quantization** is set per cell in the **AgentX cells** preset: resident c1 passes no online quantization config, preserving the BF16 target weights required by specialized decode. Every other cell quantizes only the KDA `in_proj_qkvgfab` projections to FP8 per-channel. Routed experts retain their native MXFP4 weights. The DSpark draft remains BF16 with FP8 KV: vLLM builds the named draft from its own unquantized config, so removing c1's target quantization does not change the draft's dtype.\n- **DCP8** (`--decode-context-parallel-size 8`, `--attention-backend ROCM_AITER_MLA`, `--dcp-comm-backend a2a`, `ROCM_AITER_FA` prefill, and the direct DCP ops off) is part of the **AgentX cells** preset; this preset serves without DCP.\n- The **Simple** KV-offload option emits the connector the lane uses (`SimpleCPUOffloadConnector`, `kv_both`, `lazy_offload: false`).\n\n**MoE environment.** The lane sets `VLLM_ROCM_USE_AITER_MOE_SITUV2=1`, as the AMD profile does, plus `AITER_BF16_FP8_MOE_BOUND=0` and `AITER_FLYDSL_STAGE2_FP8=1`. The SiTUv2 path does not read `AITER_BF16_FP8_MOE_BOUND`. `AITER_FLYDSL_STAGE2_FP8=1` makes the FlyDSL MoE down projection write its per-route outputs as MXFP8 (FP8 values with E8M0 scales) instead of BF16 before they are summed. It affects only the target's MoE layers; the DSpark draft's MLP is dense.\n\n**QuickReduce.** The lane also enables INT4 QuickReduce (`VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4`) with `VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB=16`, so only tensor-parallel all-reduces of 16 MiB or more use the INT4 codec. INT4 all-reduce is lossy, so it is not in the builder. The DSpark draft's all-reduces carry only its draft-step tokens (at most 600 tokens × 7,168 BF16 values, about 8.2 MiB, at concurrency 60), and its context projection has no all-reduce, so the 16 MiB floor keeps the draft out of the INT4 path in every lane cell.\n\nThe lane's ten per-concurrency cells, with DCP8, the lane's draft and the qualified image, are the [AgentX cells](#preset-mi355x_agentx_cells) preset.\n", "json": "/moonshotai/Kimi-K3/presets/mi355x_agentx.json" }, "mi355x_agentx_cells": { "label": "AgentX cells", "hardware": "mi355x", "strategy": "single_node_tp", "workload": "Agentic (InferenceX AgentX)", "description": "The lane's ten per-concurrency cells on one MI355X node: TP8 with DCP8 and the DSpark draft, GPU-resident or with host KV offload.", "source": { "label": "InferenceX #4473", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4473", "key": "kimik3-fp4-mi355x-vllm-agentic-mtp", "revision": "8737db4e8a1fae1f2e9cd03d04f2e03d6bf90145" }, "image": "vllm/vllm-openai-rocm:nightly-rocm100-df6603f0583375f244d1ac6d36265462b7490392", "select": { "frontend": "python", "features": { "spec_decoding": true, "text_only": true } }, "args": [ "--gpu-memory-utilization", "0.9", "--max-model-len", "1048576", "--enable-prefix-caching", "--kv-cache-dtype", "fp8", "--attention-config", "{\"mla_prefill_backend\":\"ROCM_AITER_FA\"}", "--stream-interval", "10", "--decode-context-parallel-size", "8", "--dcp-comm-backend", "a2a", "--attention-backend", "ROCM_AITER_MLA" ], "env": { "VLLM_ROCM_AITER_MLA_ASM_PADDING": "asm", "VLLM_ROCM_AITER_MLA_DCP_VERIFY": "asm", "AITER_FLYDSL_STAGE2_FP8": "1", "VLLM_USE_DIRECT_DCP_A2A": "0", "VLLM_USE_DIRECT_DCP_Q_GATHER": "0", "VLLM_USE_DIRECT_DCP_KV_GATHER": "0" }, "points": { "r1": { "label": "1 (resident)", "concurrency": 1, "select": { "kv_offload": "off" }, "args": [ "--max-num-seqs", "1", "--max-num-batched-tokens", "16384", "--additional-config", "{\"decode_backend\":\"latency_specialized\",\"kda_prefill_backend\":\"fused\"}" ], "env": { "VLLM_USE_V2_MODEL_RUNNER": "1" }, "json_args": { "--compilation-config": { "mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "max_cudagraph_capture_size": 16, "cudagraph_capture_sizes": [ 2, 4, 6, 8, 10, 12, 14, 16 ] } } }, "r4": { "label": "4 (resident)", "concurrency": 4, "select": { "kv_offload": "off" }, "args": [ "--max-num-seqs", "2", "--max-num-batched-tokens", "8192", "--additional-config", "{\"kda_prefill_backend\":\"fused\"}", "--quantization-config", "{\"targets\":{\"*in_proj_qkvgfab\":\"fp8_per_channel\"}}" ], "json_args": { "--compilation-config": { "mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "max_cudagraph_capture_size": 16, "cudagraph_capture_sizes": [ 2, 4, 6, 8, 10, 12, 14, 16 ] } } }, "o4": { "label": "4 (offload)", "concurrency": 4, "select": { "kv_offload": "simple" }, "args": [ "--max-num-seqs", "2", "--max-num-batched-tokens", "8192", "--additional-config", "{\"kda_prefill_backend\":\"fused\"}", "--quantization-config", "{\"targets\":{\"*in_proj_qkvgfab\":\"fp8_per_channel\"}}" ], "env": { "PYTHONHASHSEED": "42" }, "json_args": { "--speculative-config": { "num_speculative_tokens": 5 }, "--compilation-config": { "mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "max_cudagraph_capture_size": 12, "cudagraph_capture_sizes": [ 2, 4, 6, 8, 10, 12 ] }, "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": 224875000000 } } } }, "o10": { "label": "10 (offload)", "concurrency": 10, "select": { "kv_offload": "simple" }, "args": [ "--max-num-seqs", "6", "--max-num-batched-tokens", "8192", "--additional-config", "{\"kda_prefill_backend\":\"fused\"}", "--quantization-config", "{\"targets\":{\"*in_proj_qkvgfab\":\"fp8_per_channel\"}}" ], "env": { "PYTHONHASHSEED": "42" }, "json_args": { "--speculative-config": { "num_speculative_tokens": 4 }, "--compilation-config": { "mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "max_cudagraph_capture_size": 30, "cudagraph_capture_sizes": [ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30 ] }, "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": 224875000000 } } } }, "o16": { "label": "16 (offload)", "concurrency": 16, "select": { "kv_offload": "simple" }, "args": [ "--max-num-seqs", "10", "--max-num-batched-tokens", "8192", "--additional-config", "{\"kda_prefill_backend\":\"fused\"}", "--quantization-config", "{\"targets\":{\"*in_proj_qkvgfab\":\"fp8_per_channel\"}}" ], "env": { "PYTHONHASHSEED": "42" }, "json_args": { "--speculative-config": { "num_speculative_tokens": 4 }, "--compilation-config": { "mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "max_cudagraph_capture_size": 50, "cudagraph_capture_sizes": [ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50 ] }, "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": 224875000000 } } } }, "o24": { "label": "24 (offload)", "concurrency": 24, "select": { "kv_offload": "simple" }, "args": [ "--max-num-seqs", "16", "--max-num-batched-tokens", "8192", "--additional-config", "{\"kda_prefill_backend\":\"fused\"}", "--quantization-config", "{\"targets\":{\"*in_proj_qkvgfab\":\"fp8_per_channel\"}}" ], "env": { "PYTHONHASHSEED": "42" }, "json_args": { "--speculative-config": { "num_speculative_tokens": 4 }, "--compilation-config": { "mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "max_cudagraph_capture_size": 80, "cudagraph_capture_sizes": [ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80 ] }, "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": 224875000000 } } } }, "o32": { "label": "32 (offload)", "concurrency": 32, "select": { "kv_offload": "simple" }, "args": [ "--max-num-seqs", "24", "--max-num-batched-tokens", "8192", "--additional-config", "{\"kda_prefill_backend\":\"fused\"}", "--quantization-config", "{\"targets\":{\"*in_proj_qkvgfab\":\"fp8_per_channel\"}}" ], "env": { "PYTHONHASHSEED": "42" }, "json_args": { "--speculative-config": { "num_speculative_tokens": 4 }, "--compilation-config": { "mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "max_cudagraph_capture_size": 120, "cudagraph_capture_sizes": [ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 100, 102, 104, 106, 108, 110, 112, 114, 116, 118, 120 ] }, "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": 224875000000 } } } }, "o48": { "label": "48 (offload)", "concurrency": 48, "select": { "kv_offload": "simple" }, "args": [ "--max-num-seqs", "96", "--max-num-batched-tokens", "8192", "--additional-config", "{\"kda_prefill_backend\":\"fused\"}", "--quantization-config", "{\"targets\":{\"*in_proj_qkvgfab\":\"fp8_per_channel\"}}" ], "env": { "PYTHONHASHSEED": "42" }, "json_args": { "--speculative-config": { "num_speculative_tokens": 4 }, "--compilation-config": { "mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "max_cudagraph_capture_size": 480, "cudagraph_capture_sizes": [ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 100, 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, 122, 124, 126, 128, 130, 132, 134, 136, 138, 140, 142, 144, 146, 148, 150, 152, 154, 156, 158, 160, 162, 164, 166, 168, 170, 172, 174, 176, 178, 180, 182, 184, 186, 188, 190, 192, 194, 196, 198, 200, 202, 204, 206, 208, 210, 212, 214, 216, 218, 220, 222, 224, 226, 228, 230, 232, 234, 236, 238, 240, 242, 244, 246, 248, 250, 252, 254, 256, 258, 260, 262, 264, 266, 268, 270, 272, 274, 276, 278, 280, 282, 284, 286, 288, 290, 292, 294, 296, 298, 300, 302, 304, 306, 308, 310, 312, 314, 316, 318, 320, 322, 324, 326, 328, 330, 332, 334, 336, 338, 340, 342, 344, 346, 348, 350, 352, 354, 356, 358, 360, 362, 364, 366, 368, 370, 372, 374, 376, 378, 380, 382, 384, 386, 388, 390, 392, 394, 396, 398, 400, 402, 404, 406, 408, 410, 412, 414, 416, 418, 420, 422, 424, 426, 428, 430, 432, 434, 436, 438, 440, 442, 444, 446, 448, 450, 452, 454, 456, 458, 460, 462, 464, 466, 468, 470, 472, 474, 476, 478, 480 ] }, "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": 224875000000 } } } }, "o52": { "label": "52 (offload)", "concurrency": 52, "select": { "kv_offload": "simple" }, "args": [ "--max-num-seqs", "104", "--max-num-batched-tokens", "8192", "--additional-config", "{\"kda_prefill_backend\":\"fused\"}", "--quantization-config", "{\"targets\":{\"*in_proj_qkvgfab\":\"fp8_per_channel\"}}" ], "env": { "PYTHONHASHSEED": "42" }, "json_args": { "--speculative-config": { "num_speculative_tokens": 4 }, "--compilation-config": { "mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "max_cudagraph_capture_size": 520, "cudagraph_capture_sizes": [ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 100, 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, 122, 124, 126, 128, 130, 132, 134, 136, 138, 140, 142, 144, 146, 148, 150, 152, 154, 156, 158, 160, 162, 164, 166, 168, 170, 172, 174, 176, 178, 180, 182, 184, 186, 188, 190, 192, 194, 196, 198, 200, 202, 204, 206, 208, 210, 212, 214, 216, 218, 220, 222, 224, 226, 228, 230, 232, 234, 236, 238, 240, 242, 244, 246, 248, 250, 252, 254, 256, 258, 260, 262, 264, 266, 268, 270, 272, 274, 276, 278, 280, 282, 284, 286, 288, 290, 292, 294, 296, 298, 300, 302, 304, 306, 308, 310, 312, 314, 316, 318, 320, 322, 324, 326, 328, 330, 332, 334, 336, 338, 340, 342, 344, 346, 348, 350, 352, 354, 356, 358, 360, 362, 364, 366, 368, 370, 372, 374, 376, 378, 380, 382, 384, 386, 388, 390, 392, 394, 396, 398, 400, 402, 404, 406, 408, 410, 412, 414, 416, 418, 420, 422, 424, 426, 428, 430, 432, 434, 436, 438, 440, 442, 444, 446, 448, 450, 452, 454, 456, 458, 460, 462, 464, 466, 468, 470, 472, 474, 476, 478, 480, 482, 484, 486, 488, 490, 492, 494, 496, 498, 500, 502, 504, 506, 508, 510, 512, 514, 516, 518, 520 ] }, "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": 224875000000 } } } }, "o60": { "label": "60 (offload)", "concurrency": 60, "select": { "kv_offload": "simple" }, "args": [ "--max-num-seqs", "120", "--max-num-batched-tokens", "8192", "--additional-config", "{\"kda_prefill_backend\":\"fused\"}", "--quantization-config", "{\"targets\":{\"*in_proj_qkvgfab\":\"fp8_per_channel\"}}" ], "env": { "PYTHONHASHSEED": "42" }, "json_args": { "--speculative-config": { "num_speculative_tokens": 4 }, "--compilation-config": { "mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "max_cudagraph_capture_size": 600, "cudagraph_capture_sizes": [ 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 100, 102, 104, 106, 108, 110, 112, 114, 116, 118, 120, 122, 124, 126, 128, 130, 132, 134, 136, 138, 140, 142, 144, 146, 148, 150, 152, 154, 156, 158, 160, 162, 164, 166, 168, 170, 172, 174, 176, 178, 180, 182, 184, 186, 188, 190, 192, 194, 196, 198, 200, 202, 204, 206, 208, 210, 212, 214, 216, 218, 220, 222, 224, 226, 228, 230, 232, 234, 236, 238, 240, 242, 244, 246, 248, 250, 252, 254, 256, 258, 260, 262, 264, 266, 268, 270, 272, 274, 276, 278, 280, 282, 284, 286, 288, 290, 292, 294, 296, 298, 300, 302, 304, 306, 308, 310, 312, 314, 316, 318, 320, 322, 324, 326, 328, 330, 332, 334, 336, 338, 340, 342, 344, 346, 348, 350, 352, 354, 356, 358, 360, 362, 364, 366, 368, 370, 372, 374, 376, 378, 380, 382, 384, 386, 388, 390, 392, 394, 396, 398, 400, 402, 404, 406, 408, 410, 412, 414, 416, 418, 420, 422, 424, 426, 428, 430, 432, 434, 436, 438, 440, 442, 444, 446, 448, 450, 452, 454, 456, 458, 460, 462, 464, 466, 468, 470, 472, 474, 476, 478, 480, 482, 484, 486, 488, 490, 492, 494, 496, 498, 500, 502, 504, 506, 508, 510, 512, 514, 516, 518, 520, 522, 524, 526, 528, 530, 532, 534, 536, 538, 540, 542, 544, 546, 548, 550, 552, 554, 556, 558, 560, 562, 564, 566, 568, 570, 572, 574, 576, 578, 580, 582, 584, 586, 588, 590, 592, 594, 596, 598, 600 ] }, "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": 224875000000 } } } } }, "runbook": "The single-node MI355X TP8 AgentX configuration follows [InferenceX #4473](https://github.com/SemiAnalysisAI/InferenceX/pull/4473): its [master entry](https://github.com/SemiAnalysisAI/InferenceX/blob/8737db4e8a1fae1f2e9cd03d04f2e03d6bf90145/inferencex-e2e/configs/amd-master.yaml#L766) and [lane recipe](https://github.com/SemiAnalysisAI/InferenceX/blob/8737db4e8a1fae1f2e9cd03d04f2e03d6bf90145/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3/vllm/mi355x-fp4-mtp/agentic.yaml) at `8737db4`. It serves text with tool-call and reasoning parsers, using DCP8/A2A at every point. There is no concurrency-2 point.\n\n**Per-concurrency tuning.** The Point row renders the lane's ten cells. Every cell requests `FULL_AND_PIECEWISE` with breakable CUDA graphs disabled (`VLLM_USE_BREAKABLE_CUDAGRAPH=0`); because Kimi-K3 has no `torch.compile` submodule, vLLM resolves that mode to `FULL_DECODE_ONLY` at load. The [AgentX TP8](#preset-mi355x_agentx) preset keeps the builder's general values for any concurrency.\n\nEvery cell adds `kda_prefill_backend: fused` in `--additional-config`. Only **resident c1** also sets `decode_backend: latency_specialized` and `VLLM_USE_V2_MODEL_RUNNER=1`. On the pinned image, this [MI355X specialized path](https://github.com/vllm-project/vllm/pull/60683) requires TP8, V2, gfx950, no KV connector and BF16 target projections. It handles pure speculative-verification batches of at most eight rows (c1 with K7); other steps use the normal layers. These lane-specific settings stay out of generic MI355X defaults.\n\nThe `resident` profile keeps KV on GPU; `offload` uses `SimpleCPUOffloadConnector` with 224.875 GB of host DRAM per rank. `--max-num-seqs` is 2 at c4 in both profiles, then 6/10/16/24 at offload c10/c16/c24/c32. Resident c1 stays at 1; offload c48/c52/c60 stay at twice concurrency. These smaller c4–c32 batches target p90 interactivity and can trade off queueing, TTFT and end-to-end latency. Every cell captures every even CUDA-graph size from 2 through `max-num-seqs × (1 + draft tokens)`, except c1 whose maximum remains 16. Agentic benchmark runs replace block rejection with the golden acceptance length; synthetic acceptance is not suitable for serving or accuracy evaluation.\n\n**Image.** This preset pins the builder and the commands below to `vllm/vllm-openai-rocm:nightly-rocm100-df6603f0583375f244d1ac6d36265462b7490392` (vLLM `df6603f0`), the image qualified by #4393, reused by #4473 and carrying #60683. The compatibility limits above refer to this pinned runtime.\n\n### Reproducing one lane cell\n\nRun each cell in a fresh shell inside the pinned image. This real-verification form is for serving and accuracy evaluation; `PROFILE` distinguishes the two concurrency-4/DCP8 cells:\n\n```bash\nPROFILE=resident\nCLIENTS=1\ncase \"$PROFILE:$CLIENTS\" in\n resident:1) K=7; BATCHED=16384; OFFLOAD=0 ;;\n resident:4) K=7; BATCHED=8192; OFFLOAD=0 ;;\n offload:4) K=5; BATCHED=8192; OFFLOAD=1 ;;\n offload:10|offload:16|offload:24|offload:32|offload:48|offload:52|offload:60) K=4; BATCHED=8192; OFFLOAD=1 ;;\n *) echo \"PROFILE=$PROFILE CLIENTS=$CLIENTS is not a lane cell\" >&2; exit 1 ;;\nesac\n\nexport VLLM_ROCM_USE_AITER=1\nexport SAFETENSORS_FAST_GPU=1\nexport VLLM_ROCM_USE_AITER_MOE_SITUV2=1\nexport AITER_BF16_FP8_MOE_BOUND=0\nexport AITER_FLYDSL_STAGE2_FP8=1\nexport VLLM_ROCM_AITER_MLA_ASM_PADDING=asm\nexport VLLM_ROCM_AITER_MLA_DCP_VERIFY=asm\nexport VLLM_USE_DIRECT_DCP_A2A=0\nexport VLLM_USE_DIRECT_DCP_Q_GATHER=0\nexport VLLM_USE_DIRECT_DCP_KV_GATHER=0\n# Lossy INT4-quantized all-reduce for messages of 16 MiB or more: not in the builder.\nexport VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4\nexport VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB=16\n# 1.56 TB of weights from shared storage, and long agentic prefills.\nexport VLLM_ENGINE_READY_TIMEOUT_S=7200\nexport VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200\nexport PYTHONNOUSERSITE=1\n# Required on the qualification nodes' MEC firmware 38; fixed in firmware 177.\nexport HSA_NO_SCRATCH_RECLAIM=1\n\nGRAPH_MODE=FULL_AND_PIECEWISE\nexport VLLM_USE_BREAKABLE_CUDAGRAPH=0\nMAX_SEQS=$((CLIENTS == 1 ? 1 : 2 * CLIENTS))\ncase \"$CLIENTS\" in\n 4) MAX_SEQS=2 ;;\n 10) MAX_SEQS=6 ;;\n 16) MAX_SEQS=10 ;;\n 24) MAX_SEQS=16 ;;\n 32) MAX_SEQS=24 ;;\nesac\nMAX_CAPTURE=$(((CLIENTS == 1 ? 2 : MAX_SEQS) * (1 + K)))\nCOMPILATION=\"{\\\"mode\\\":3,\\\"cudagraph_mode\\\":\\\"$GRAPH_MODE\\\",\\\"max_cudagraph_capture_size\\\":$MAX_CAPTURE,\\\"custom_ops\\\":[\\\"+fused_rms_norm_gated\\\"],\\\"cudagraph_capture_sizes\\\":[$(seq -s, 2 2 \"$MAX_CAPTURE\")]}\"\n\nEXTRA=(--additional-config '{\"kda_prefill_backend\":\"fused\"}')\nif [ \"$PROFILE:$CLIENTS\" = resident:1 ]; then\n export VLLM_USE_V2_MODEL_RUNNER=1\n EXTRA=(--additional-config '{\"decode_backend\":\"latency_specialized\",\"kda_prefill_backend\":\"fused\"}')\nelse\n EXTRA+=(--quantization-config '{\"targets\":{\"*in_proj_qkvgfab\":\"fp8_per_channel\"}}')\nfi\nif [ \"$K\" -gt 0 ]; then\n EXTRA+=(--speculative-config \"{\\\"model\\\":\\\"Inferact/Kimi-K3-DSpark\\\",\\\"num_speculative_tokens\\\":$K,\\\"method\\\":\\\"dspark\\\",\\\"attention_backend\\\":\\\"ROCM_AITER_MLA\\\",\\\"kv_cache_dtype\\\":\\\"fp8\\\",\\\"draft_sample_method\\\":\\\"probabilistic\\\",\\\"rejection_sample_method\\\":\\\"block\\\"}\")\nfi\nif [ \"$OFFLOAD\" -eq 1 ]; then\n # The lane's node gives the connector 1799 GB of host DRAM (224875000000 bytes\n # per rank); size cpu_bytes_to_use_per_rank to your host. The fixed hash seed\n # makes identical prefixes map to the same block keys on every rank.\n export PYTHONHASHSEED=42\n EXTRA+=(--kv-transfer-config '{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use_per_rank\":224875000000,\"lazy_offload\":false}}')\nfi\nvllm serve moonshotai/Kimi-K3 \\\n --trust-remote-code \\\n --moe-backend auto \\\n --tensor-parallel-size 8 \\\n --decode-context-parallel-size 8 \\\n --dcp-comm-backend a2a \\\n --gpu-memory-utilization 0.9 \\\n --language-model-only \\\n --max-num-seqs \"$MAX_SEQS\" \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 \\\n --reasoning-parser kimi_k3 \\\n --max-model-len 1048576 \\\n --enable-prefix-caching \\\n --kv-cache-dtype fp8 \\\n --stream-interval 10 \\\n --max-num-batched-tokens \"$BATCHED\" \\\n --attention-config '{\"mla_prefill_backend\":\"ROCM_AITER_FA\"}' \\\n --compilation-config \"$COMPILATION\" \\\n \"${EXTRA[@]}\"\n```\n\nOutside this preset, the generic builder's **Simple** option uses 220 GiB per rank and does not set `PYTHONHASHSEED`.\n\n### Throughput measurement only (synthetic acceptance)\n\nThe lane's throughput runs replace block verification with synthetic acceptance, so every concurrency point is measured at InferenceX's golden acceptance length for this draft. Only the `--speculative-config` changes:\n\n```bash\n--speculative-config \"{\\\"model\\\":\\\"Inferact/Kimi-K3-DSpark\\\",\\\"num_speculative_tokens\\\":$K,\\\"method\\\":\\\"dspark\\\",\\\"attention_backend\\\":\\\"ROCM_AITER_MLA\\\",\\\"kv_cache_dtype\\\":\\\"fp8\\\",\\\"draft_sample_method\\\":\\\"probabilistic\\\",\\\"rejection_sample_method\\\":\\\"synthetic\\\",\\\"synthetic_acceptance_length\\\":$AL}\"\n```\n\n| Draft tokens | Golden acceptance length | Lane cells |\n| ---: | ---: | --- |\n| 7 | 3.84 | resident concurrency 1 and 4, DCP8 |\n| 5 | 3.62 | offload concurrency 4, DCP8 |\n| 4 | 3.36 | dcp8 concurrency 10, 16, 24, 32, 48, 52 and 60 |\n\nThese lengths come from [`kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml`](https://github.com/SemiAnalysisAI/InferenceX/blob/8737db4e8a1fae1f2e9cd03d04f2e03d6bf90145/inferencex-e2e/infx/golden_al_distribution/kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml): SPEED-Bench coding prompts with thinking on, measured on B300 for `Inferact/Kimi-K3-DSpark` with probabilistic drafting and block verification. They hold only for that draft at those draft lengths, not for `RedHatAI/Kimi-K3-speculator.dspark`. Synthetic acceptance skips real verification: never use it to serve users or to evaluate accuracy.\n\n### Evidence and limits\n\n- InferenceX [run 40132624130](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40132624130) passed all ten performance jobs at source `684d891612596e4ed9dbdd2f402865e288214815`. These throughput measurements use synthetic acceptance. The selected matrix, image and fully bound lane configurations match `8737db4`; this is reused source-run evidence, not a rerun of that head.\n- The run's ten accuracy jobs use real verification and cover nine distinct lane cells: offload c4 was not evaluated, and c60 has both vendor and lm-eval jobs. See the [accuracy collection](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40132624130/job/121058737731). Performance collection and comparison were skipped in this unofficial run; no across-the-board throughput or latency gain is claimed, and these results have not been reproduced locally for this recipe.\n- Model policy (vLLM as the native engine, `Inferact/Kimi-K3-DSpark` as the agreed draft): [`MODELS.md`](https://github.com/SemiAnalysisAI/InferenceX/blob/8737db4e8a1fae1f2e9cd03d04f2e03d6bf90145/inferencex-e2e/docs/MODELS.md).\n", "json": "/moonshotai/Kimi-K3/presets/mi355x_agentx_cells.json" } }, "guide": "## Overview\n\nKimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model (16 of 896 experts active per token) built on the Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), with a 1M-token context window and native vision. \n\n## Prerequisites\n\n- **vLLM:** Use the vllm/vllm-openai:kimi-k3 docker\n- **CUDA:** The kimi-k3 image ships as a CUDA 13 (cu130) build only — there is no -cu129 tag, and the K3-enabled wheels are not on the cu129 nightly index. The host needs an r580+ NVIDIA driver; on a CUDA 12.9 (r575) host, upgrade the driver or build vLLM from the K3 branch against cu129 PyTorch yourself.\n- **Hardware:** At least 8x GB300. Multi-node for real production traffic.\n- **ROCm:** At least 8x MI355X/MI350X. On MI355X the command builder uses the ROCm 10.0 nightly image `vllm/vllm-openai-rocm:nightly-rocm100`; the day-0 `vllm/vllm-openai-rocm:kimi-k3` image predates the DSpark and MoE fixes that path needs (see the [MI355X AgentX preset](#preset-mi355x_agentx)).\n\n## Client Usage\n\nOnce the vLLM server is running, consume it via the OpenAI-compatible API:\n\n```python\nimport time\nfrom openai import OpenAI\n\nclient = OpenAI(\n api_key=\"EMPTY\",\n base_url=\"http://localhost:8000/v1\",\n timeout=3600\n)\n\nmessages = [\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"type\": \"image_url\",\n \"image_url\": {\n \"url\": \"https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png\"\n }\n },\n {\n \"type\": \"text\",\n \"text\": \"Read all the text in the image.\"\n }\n ]\n }\n]\n\nstart = time.time()\nresponse = client.chat.completions.create(\n model=\"moonshotai/Kimi-K3\",\n messages=messages,\n max_tokens=2048\n)\nprint(f\"Response costs: {time.time() - start:.2f}s\")\nprint(f\"Generated text: {response.choices[0].message.content}\")\n```\n\n## Notes\n\n- **Cross node communitcation**: Use `--all2all-backend deepep_v2` for RDMA and `--all2all-backend flashinfer_nvlink_one_sided` for NVLink.\n- **MoE backend**: Recommend to use `deep_gemm_mega_moe` for DEP deployments with cross-node NVLink e.g. GB200 and GB300. Note that DeepGEMM MegaMoE is not compatible with cross-node RDMA.\n- **Rust Frontend**: Selected by default in the command builder. Switch to Python if you encounter unsupported features or compatibility issues.\n- **Model Runner v2**: `VLLM_USE_V2_MODEL_RUNNER=1`: Model Runner v2 fully supports this model and can be enabled if needed.\n- **Tool calling**: K3 occasionally emit a tool-call format its own parser doesn't expect. Suggest to run do schema validation and retry.\n- **AMD (MI355X / MI350X, CDNA4 gfx950)**: `VLLM_ROCM_USE_AITER_MOE_SITUV2=1` (set above) enables AITER's SiTUv2 a4w4 FlyDSL MoE path on builds that carry vllm-project/vllm#53940 (ROCm nightlies since 2026-09-16). v0.30.0 predates it: there the variable is ignored and the MoE runs the default a16w4 path. Unset it or set `0` for the default a16w4 path. On builds with #53940, `VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4` is a deprecated alias for the same a4w4 path and vLLM removes `AITER_SITUV2_A8W4` at startup while the toggle is on; on v0.30.0 and older, `VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4=1` selects the a8w4 path instead. Do not set `AITER_SITUV2_A8W4=1` by itself: vLLM does not derive its MoE weight layout from that AITER flag (vllm-project/vllm#53954 is open), so the two can disagree and corrupt output without an error.\n- **max-model-len**: Adjust max-model-len for different benchmark scenarios for best performance.\n- **RDMA**: If RDMA is enabled, set `UCX_TLS=\"rc,cuda_copy\"` to make sure KV Cache transfer goes through RDMA.\n- **MNNVL environments** (GB200/GB300 NVL): recommend adding `NCCL_MNNVL_ENABLE=1`, `NCCL_CUMEM_ENABLE=1`, and `NCCL_NVLS_ENABLE=1`.\n- **mlx5 dmabuf registration failures**: if engine init fails with \"NCCL error: unhandled system error\" and the log shows `mlx5dv_reg_dmabuf_mr` errno 524, the kernel/driver lacks mlx5 dmabuf support (NCCL 2.28 registers dmabuf by default). Set `NCCL_DMABUF_ENABLE=0` to fall back to `nvidia_peermem` (must be loaded on the nodes) — still GPUDirect RDMA.\n- **FP8 KV**: If FP8 KV cache is needed, please also add `--attention-config '{\"use_prefill_query_quantization\":true,\"mla_prefill_backend\":\"TOKENSPEED_MLA\"}'` when serving vLLM. `TOKENSPEED_MLA` is what the Blackwell profile above emits (paired with `--attention-backend TOKENSPEED_MLA` for decode), but it is not the only supported choice: vLLM 0.27.1 registers three MLA prefill backends — `FLASHINFER`, `TRTLLM_RAGGED` and `TOKENSPEED_MLA` (see `vllm/v1/attention/backends/mla/prefill/registry.py`) — and any of them may be substituted for `mla_prefill_backend` alongside `use_prefill_query_quantization`. Which one wins depends on the input-length distribution, so benchmark before switching. Those backends do not run on ROCm: on MI355X the builder pairs FP8 KV with `--attention-config '{\"mla_prefill_backend\":\"ROCM_AITER_FA\"}'` instead.\n- **Prefix-match unit**: K3 is a hybrid model: its MLA attention layers and KDA (Mamba-like) layers form two KV-cache groups under the hybrid KV-cache manager. With prefix caching on, vLLM pads the attention block size up to match the Mamba state page, so both groups resolve to one (large) block size and the default prefix-cache hit boundary lands on that block — very coarse. The Blackwell profile sets `--prefix-match-unit 128`: the inflation formula always yields a multiple of 128, so 128 divides whatever block size results, and it aligns with the MLA kernel's native block boundary, giving finer prefix-hit granularity. The value is only effective when prefix caching (or a KV connector) is active; it is a no-op otherwise, and it is safe under all tensor-parallel layouts.\n- **Decode context parallelism (DCP)**: the qualified DCP layouts are part of the presets that ran them: DCP8 on the GB200 aggregated TP8 x PP2 and GB300 disaggregated TP8 pools, DCP16 on the GB200 TP16 disaggregated pools, DCP16/DCP8 at the GB300 low-latency points, and DCP8 in the MI355X AgentX cells. Each pairs `--decode-context-parallel-size` (a divisor of the tensor-parallel size) with `--dcp-comm-backend a2a` and an MLA attention backend that supports it (`TOKENSPEED_MLA` with `mla_prefill_backend: TRTLLM_RAGGED` on Blackwell, `ROCM_AITER_MLA` with `mla_prefill_backend: ROCM_AITER_FA` on MI355X). On builds that carry them, the `VLLM_USE_DIRECT_DCP_A2A`, `VLLM_USE_DIRECT_DCP_Q_GATHER` and `VLLM_USE_DIRECT_DCP_KV_GATHER` knobs default to auto; set them to `1` to pin the direct DCP ops or `0` to disable them, as the MI355X cells do.\n- Ascend W4A8 checkpoint (ModelScope): https://www.modelscope.cn/models/Eco-Tech/Kimi-K3-w4a8\n- vLLM Ascend tutorial: https://docs.vllm.ai/projects/ascend/en/v0.23.0/tutorials/models/Kimi-K3.html\n\n## AMD MI355X: disaggregated prefill and decode\n\nThis example uses one TP8/DCP8 prefill worker and one TP8/DCP8 decode worker on separate 8-GPU MI355X nodes. MoRIIO transfers the KV cache in READ mode; `MultiConnector` combines it with `SimpleCPUOffloadConnector` on prefill only. Both roles use the same target and DSpark draft checkpoints and FP8 KV.\n\nThese standalone commands are based on the 1P1D c48 configuration in [InferenceX#4391](https://github.com/SemiAnalysisAI/InferenceX/pull/4391), with real block verification for serving and accuracy evaluation. They are separate from the command builder's NIXL PD configuration.\n\n### Prerequisites\n\n- Make [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) and [Inferact/Kimi-K3-DSpark](https://huggingface.co/Inferact/Kimi-K3-DSpark) available inside every worker container. The commands below use `/models/Kimi-K3` and `/models/Inferact-Kimi-K3-DSpark` as example container paths; adjust them to match your deployment. Use identical checkpoint revisions on all workers.\n- Use the worker and router images below, with the same worker image build on every node. The worker image contains the required vLLM support; no source patches are needed.\n- Expose all eight GPUs, `/dev/infiniband`, host networking, and sufficient shared memory to each worker container. Allow locked memory for RDMA. The container's verbs provider must be compatible with the host NIC and driver.\n- The prefill offload budget is 2,399 GB of host DRAM **across all eight ranks**, in addition to weight-loading and runtime memory. Reduce `CPU_OFFLOAD_BYTES` if the host cannot accommodate that allocation; the documented configurations use `2399000000000`.\n- Set `SOCKET_IFNAME` and `RDMA_DEVICES` to your network interface and HCA list. `ROUTER_IP` and each `WORKER_IP` must be reachable addresses, not `0.0.0.0`. Allow router TCP ports 8000 and 36367, worker HTTP port 6100, and MoRIIO handshake/notification ports 26000–26007 and 27000–27007, plus the site's RDMA traffic.\n\n```bash\nWORKER_IMAGE=vllm/vllm-openai-rocm:nightly\nROUTER_IMAGE=vllm/vllm-router:nightly\n```\n\n### Start the router\n\nStart one router before the workers. Discovery registers workers automatically; do not add static `--prefill` or `--decode` endpoints.\n\n```bash\ndocker run --rm --network host --entrypoint vllm-router \"$ROUTER_IMAGE\" \\\n --vllm-pd-disaggregation \\\n --kv-connector moriio \\\n --vllm-discovery-address 0.0.0.0:36367 \\\n --host 0.0.0.0 --port 8000 \\\n --worker-startup-timeout-secs 7200 \\\n --policy consistent_hash \\\n --prefill-policy consistent_hash \\\n --decode-policy consistent_hash \\\n --log-level info\n```\n\n### Start the workers\n\nSave the following as `kimi-k3-pd.sh` on each worker. Run it with Bash **inside the worker image**, with the devices and model paths described above.\n\n```bash\n#!/usr/bin/env bash\nset -euo pipefail\n\nROLE=${1:?Usage: bash kimi-k3-pd.sh prefill|decode}\n: \"${ROUTER_IP:?Set the router reachable IP}\"\n: \"${WORKER_IP:?Set this worker reachable IP}\"\n: \"${SOCKET_IFNAME:?Set the host network interface}\"\n: \"${RDMA_DEVICES:?Set the comma-separated HCA list}\"\nCPU_OFFLOAD_BYTES=${CPU_OFFLOAD_BYTES:-2399000000000}\n\nexport NCCL_SOCKET_IFNAME=\"$SOCKET_IFNAME\" GLOO_SOCKET_IFNAME=\"$SOCKET_IFNAME\"\nexport NCCL_IB_HCA=\"$RDMA_DEVICES\"\nexport VLLM_ROCM_USE_AITER=1 AITER_BF16_FP8_MOE_BOUND=0\nexport AITER_FLYDSL_STAGE2_FP8=1\nexport VLLM_ROCM_AITER_MLA_ASM_PADDING=asm VLLM_ROCM_AITER_MLA_DCP_VERIFY=asm\nexport VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4\nexport VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB=256\nexport SAFETENSORS_FAST_GPU=1\nexport VLLM_SSM_CONV_STATE_LAYOUT=DS VLLM_KV_CACHE_LAYOUT=HND\nexport NCCL_DMABUF_ENABLE=0 HSA_ENABLE_IPC_MODE_LEGACY=1 HIP_FORCE_DEV_KERNARG=1\nexport PYTHONHASHSEED=42 PYTHONNOUSERSITE=1\nexport VLLM_USE_DIRECT_DCP_A2A=0 VLLM_USE_DIRECT_DCP_Q_GATHER=0 VLLM_USE_DIRECT_DCP_KV_GATHER=0\nexport VLLM_ENGINE_READY_TIMEOUT_S=7200 VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200\nexport MORI_IO_SQ_BACKOFF_TIMEOUT_US=50000\nexport MORI_IO_QP_MAX_SEND_WR=16384 MORI_IO_QP_MAX_CQE=32768 MORI_IO_QP_MAX_SGE=2\nexport MORI_IO_TC_DISABLE=0\n\nCUSTOM_OPS='[\"+fused_rms_norm_gated\",\"+quant_fp8\",\"+grouped_topk\",\"+sparse_attn_indexer\",\"none\"]'\ncase \"$ROLE\" in\n prefill)\n MAX_SEQS=96\n BATCHED=8192\n KV_ROLE=kv_producer\n export HSA_NO_SCRATCH_RECLAIM=0\n COMPILATION=\"{\\\"mode\\\":3,\\\"cudagraph_mode\\\":\\\"NONE\\\",\\\"max_cudagraph_capture_size\\\":0,\\\"custom_ops\\\":$CUSTOM_OPS}\"\n ;;\n decode)\n MAX_SEQS=96\n BATCHED=512\n KV_ROLE=kv_consumer\n CAPTURES=\"$(seq -s, 1 128),$(seq -s, 130 10 240)\"\n COMPILATION=\"{\\\"mode\\\":3,\\\"cudagraph_mode\\\":\\\"FULL_DECODE_ONLY\\\",\\\"max_cudagraph_capture_size\\\":240,\\\"cudagraph_capture_sizes\\\":[$CAPTURES],\\\"custom_ops\\\":$CUSTOM_OPS}\"\n ;;\n *) echo \"Role must be prefill or decode\" >&2; exit 1 ;;\nesac\n\nMORI=\"{\\\"kv_connector\\\":\\\"MoRIIOConnector\\\",\\\"kv_role\\\":\\\"$KV_ROLE\\\",\\\"kv_load_failure_policy\\\":\\\"fail\\\",\\\"kv_connector_extra_config\\\":{\\\"backend\\\":\\\"rdma\\\",\\\"read_mode\\\":true,\\\"proxy_ip\\\":\\\"$ROUTER_IP\\\",\\\"proxy_ping_port\\\":\\\"36367\\\",\\\"host_ip\\\":\\\"$WORKER_IP\\\",\\\"http_port\\\":\\\"6100\\\",\\\"handshake_port\\\":\\\"26000\\\",\\\"notify_port\\\":\\\"27000\\\"}}\"\nKV_CONFIG=$MORI\nif [ \"$ROLE\" = prefill ]; then\n CPU=\"{\\\"kv_connector\\\":\\\"SimpleCPUOffloadConnector\\\",\\\"kv_role\\\":\\\"kv_both\\\",\\\"kv_connector_extra_config\\\":{\\\"cpu_bytes_to_use\\\":$CPU_OFFLOAD_BYTES,\\\"lazy_offload\\\":false}}\"\n KV_CONFIG=\"{\\\"kv_connector\\\":\\\"MultiConnector\\\",\\\"kv_role\\\":\\\"kv_both\\\",\\\"kv_load_failure_policy\\\":\\\"fail\\\",\\\"kv_connector_extra_config\\\":{\\\"connectors\\\":[$MORI,$CPU]}}\"\nfi\n\nexec vllm serve /models/Kimi-K3 \\\n --host 0.0.0.0 --port 6100 \\\n --served-model-name moonshotai/Kimi-K3 \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --decode-context-parallel-size 8 \\\n --cp-kv-cache-interleave-size 1 \\\n --dcp-comm-backend a2a \\\n --moe-backend auto \\\n --load-format fastsafetensors \\\n --gpu-memory-utilization 0.90 \\\n --language-model-only \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 \\\n --max-model-len 1048576 \\\n --stream-interval 10 \\\n --enable-prefix-caching --prefix-match-unit 128 \\\n --kv-cache-dtype fp8 --mamba-ssm-cache-dtype bfloat16 \\\n --block-size 128 \\\n --attention-backend ROCM_AITER_MLA \\\n --attention-config '{\"mla_prefill_backend\":\"ROCM_AITER_FA\",\"use_prefill_query_quantization\":true}' \\\n --speculative-config '{\"model\":\"/models/Inferact-Kimi-K3-DSpark\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"attention_backend\":\"ROCM_AITER_MLA\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}' \\\n --max-num-seqs \"$MAX_SEQS\" \\\n --max-num-batched-tokens \"$BATCHED\" \\\n --compilation-config \"$COMPILATION\" \\\n --kv-transfer-config \"$KV_CONFIG\"\n```\n\nOn the prefill worker, replace the example addresses, interface, and HCAs with your site's values:\n\n```bash\nexport ROUTER_IP=192.0.2.10 WORKER_IP=192.0.2.11\nexport SOCKET_IFNAME=eth0 RDMA_DEVICES=rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7\nbash kimi-k3-pd.sh prefill\n```\n\nOn the decode worker:\n\n```bash\nexport ROUTER_IP=192.0.2.10 WORKER_IP=192.0.2.12\nexport SOCKET_IFNAME=eth0 RDMA_DEVICES=rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7\nbash kimi-k3-pd.sh decode\n```\n\nPrefill uses an 8,192-token batch budget without CUDA graphs; decode uses a 512-token budget with `FULL_DECODE_ONLY` graphs. Both roles use a 96-sequence limit and four DSpark draft tokens, with BF16 SSM. INT4 QuickReduce is a quantized collective optimization; validate accuracy for your workload and keep the same setting on all workers. The RDMA queue backoff is 50,000 microseconds (50 ms).\n\nPrefill scratch reclamation is enabled with `HSA_NO_SCRATCH_RECLAIM=0`, matching the c48 configurations.\n\n### Scaling and concurrency tuning\n\nFor 1P2D or 1P3D, run the decode command on additional nodes with their own `WORKER_IP`. Discovery registers each worker automatically; the port ranges can be reused on separate hosts. These are router-balanced replicas, not a cross-node TP group.\n\nFor the offload configurations, let `C` be global client concurrency, `N` the number of decode workers, and `K` the draft length:\n\n```text\nP max-num-seqs = 2 * C\nD max-num-seqs = ceil(2 * C / N)\nD graph max G = (K + 1) * C\n```\n\nThese are tuning starting points; sequence limits provide scheduling headroom. Keep P/D SSM dtypes, DCP settings, and cache layouts matched.\n\n| Topology | Global concurrency | P / per-D max sequences | Draft tokens | SSM state | P CPU offload | D graph dense / max |\n| --- | ---: | --- | ---: | --- | --- | --- |\n| 1P1D | 8 | 1 / 1 | 7 | FP32 | None | Explicit `[1, 8]` |\n| 1P1D | 48 | 96 / 96 | 4 | BF16 | 2399 GB | 128 / 240 |\n| 1P2D | 24 | 48 / 24 | 4 | BF16 | 2399 GB | 64 / 120 |\n| 1P2D | 48 | 96 / 48 | 4 | BF16 | 2399 GB | 128 / 240 |\n| 1P3D | 24 | 48 / 16 | 4 | BF16 | 2399 GB | 64 / 120 |\n\nFor c8, set both sequence limits to 1, use K7 and FP32 SSM, and remove prefill CPU offload (use MoRIIO alone). On decode only, set `VLLM_USE_V2_MODEL_RUNNER=1`, add `--additional-config '{\"decode_backend\":\"latency_specialized\"}'`, and use graph sizes `[1, 8]` with maximum 8.\n\nBenchmark concurrency sweeps and harness settings remain in the linked InferenceX configuration. Benchmark runs use its measured golden acceptance curve; the serving commands retain real block verification.\n\n### Verify routed inference\n\nWait for every worker to load and register with the router. Check each worker's `/health` endpoint and the router's `/v1/models`, then send a request through the router:\n\n```bash\ncurl --fail-with-body \"http://${ROUTER_IP}:8000/v1/chat/completions\" \\\n -H 'Content-Type: application/json' \\\n -d '{\"model\":\"moonshotai/Kimi-K3\",\"messages\":[{\"role\":\"user\",\"content\":\"What is 17 + 25?\"}],\"reasoning_effort\":\"low\",\"temperature\":0.6,\"max_tokens\":4096}'\n```\n\nConfirm that a final answer is returned, and inspect the prefill/decode logs for successful KV transfers and transfer errors. Health checks alone do not exercise RDMA. Preserve `reasoning_content` and tool calls when building multi-turn conversations, as described in the [model card](https://huggingface.co/moonshotai/Kimi-K3#6-model-usage).\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/moonshotai/Kimi-K3.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/moonshotai/Kimi-K3.yaml", "overlays": [ { "source": "vllm-ascend", "repo": "vllm-project/vllm-ascend", "ref": "main", "sha": "02ac27a19cb7676304a9ca6985ba56ce29664536", "path": "recipes/overlays/moonshotai/Kimi-K3.yaml", "edit_url": "https://github.com/vllm-project/vllm-ascend/edit/main/recipes/overlays/moonshotai/Kimi-K3.yaml" } ] } }