{ "hf_id": "moonshotai/Kimi-K2-Thinking", "meta": { "title": "Kimi-K2-Thinking", "slug": "kimi-k2-thinking", "provider": "Moonshot AI", "description": "Kimi-K2-Thinking is an advanced reasoning MoE model with native INT4 QAT weights, designed for long-horizon agent workflows interleaving chain-of-thought reasoning with tool calls.", "date_added": "2025-11-06", "date_updated": "2026-04-17", "tasks": [ "text" ], "performance_headline": "1T MoE thinking model with native INT4 QAT for 2x low-latency speed-up", "related_recipes": [ "moonshotai/Kimi-K2-Instruct", "moonshotai/Kimi-K2.5" ], "hardware": { "h200": "verified", "mi300x": "verified", "mi325x": "verified", "mi355x": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve moonshotai/Kimi-K2-Thinking \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k2 \\\n --reasoning-parser kimi_k2", "argv": [ "vllm", "serve", "moonshotai/Kimi-K2-Thinking", "--trust-remote-code", "--tensor-parallel-size", "8", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k2", "--reasoning-parser", "kimi_k2" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest moonshotai/Kimi-K2-Thinking \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k2 \\\n --reasoning-parser kimi_k2", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "moonshotai/Kimi-K2-Thinking", "--trust-remote-code", "--tensor-parallel-size", "8", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k2", "--reasoning-parser", "kimi_k2" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tep": "/moonshotai/Kimi-K2-Thinking/strategies/single_node_tep.json", "single_node_dep": "/moonshotai/Kimi-K2-Thinking/strategies/single_node_dep.json", "multi_node_tp": "/moonshotai/Kimi-K2-Thinking/strategies/multi_node_tp.json", "multi_node_tp_pp": "/moonshotai/Kimi-K2-Thinking/strategies/multi_node_tp_pp.json", "multi_node_dep": "/moonshotai/Kimi-K2-Thinking/strategies/multi_node_dep.json", "multi_node_tep": "/moonshotai/Kimi-K2-Thinking/strategies/multi_node_tep.json", "pd_cluster": "/moonshotai/Kimi-K2-Thinking/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/moonshotai/Kimi-K2-Thinking/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/moonshotai/Kimi-K2-Thinking/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/moonshotai/Kimi-K2-Thinking/hw/h200.json", "h100": "/moonshotai/Kimi-K2-Thinking/hw/h100.json", "b200": "/moonshotai/Kimi-K2-Thinking/hw/b200.json", "gb200": "/moonshotai/Kimi-K2-Thinking/hw/gb200.json", "b300": "/moonshotai/Kimi-K2-Thinking/hw/b300.json", "gb300": "/moonshotai/Kimi-K2-Thinking/hw/gb300.json", "mi300x": "/moonshotai/Kimi-K2-Thinking/hw/mi300x.json", "mi325x": "/moonshotai/Kimi-K2-Thinking/hw/mi325x.json", "mi350x": "/moonshotai/Kimi-K2-Thinking/hw/mi350x.json", "mi355x": "/moonshotai/Kimi-K2-Thinking/hw/mi355x.json" } }, "model": { "model_id": "moonshotai/Kimi-K2-Thinking", "min_vllm_version": "0.12.0", "architecture": "moe", "parameter_count": "1T", "active_parameters": "32B", "context_length": 262144, "supports_dcp": true, "base_args": [ "--trust-remote-code" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "tool_calling": { "description": "Enable Kimi K2 tool calling with the kimi_k2 tool-call parser.", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k2" ] }, "reasoning": { "description": "Kimi K2 reasoning parser for extracting chain-of-thought content.", "args": [ "--reasoning-parser", "kimi_k2" ] } }, "opt_in_features": [], "variants": { "default": { "precision": "int4", "vram_minimum_gb": 600, "description": "Native INT4 (QAT) weights on 8xH200 / 8xH20; 2x low-latency speed-up vs FP8" }, "nvfp4": { "model_id": "nvidia/Kimi-K2-Thinking-NVFP4", "precision": "nvfp4", "vram_minimum_gb": 600, "description": "NVIDIA NVFP4 quantized weights for Blackwell GPUs", "extra_args": [ "--kv-cache-dtype", "fp8" ], "json": "/nvidia/Kimi-K2-Thinking-NVFP4.json" }, "fp8": { "model_id": "RedHatAI/Kimi-K2-Thinking-FP8-Block", "precision": "fp8", "vram_minimum_gb": 1200, "description": "RedHatAI FP8 block quantization", "extra_args": [ "--kv-cache-dtype", "fp8" ], "json": "/RedHatAI/Kimi-K2-Thinking-FP8-Block.json" } }, "hardware_overrides": { "amd": { "extra_args": [ "--no-enable-prefix-caching" ], "extra_env": { "SAFETENSORS_FAST_GPU": "1" } } }, "guide": "## Overview\n\n[Kimi-K2-Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking) is an advanced\ntrillion-parameter MoE created by Moonshot AI with these highlights:\n\n- **Deep Thinking & Tool Orchestration**: End-to-end trained to interleave\n chain-of-thought reasoning with function calls, enabling autonomous research, coding,\n and writing workflows that last hundreds of steps without drift.\n- **Native INT4 Quantization**: Quantization-Aware Training (QAT) delivers lossless 2x\n speed-up in low-latency mode.\n- **Stable Long-Horizon Agency**: Maintains coherent goal-directed behavior across up to\n 200-300 consecutive tool invocations, surpassing prior models that degrade after\n 30-50 steps.\n\n## Prerequisites\n\n- **Hardware**: 8x H200 or 8x H20 GPUs\n- **vLLM**: Current stable release\n\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto\n```\n\n### AMD ROCm\n\nThe ROCm wheel requires Python 3.12, ROCm 7.0+, and glibc >= 2.35. Use the\nROCm Docker image when your host environment does not meet those requirements.\n\n```bash\nuv venv --python 3.12\nsource .venv/bin/activate\nuv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/\n```\n\n## Launching Kimi-K2-Thinking with vLLM\n\n### Low-Latency Scenarios (TP8)\n\n```bash\nvllm serve moonshotai/Kimi-K2-Thinking \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k2 \\\n --reasoning-parser kimi_k2 \\\n --trust-remote-code\n```\n\nThe `--reasoning-parser` flag specifies the parser used to extract reasoning content\nfrom the model output.\n\n### ROCm on 8x MI300X / MI325X / MI355X\n\n```bash\nexport SAFETENSORS_FAST_GPU=1\nvllm serve moonshotai/Kimi-K2-Thinking \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k2 \\\n --reasoning-parser kimi_k2 \\\n --trust-remote-code\n```\n\n### High-Throughput Scenarios (TP8+DCP8)\n\nvLLM supports [Decode Context Parallel](https://docs.vllm.ai/en/latest/serving/context_parallel_deployment.html#decode-context-parallel),\nwhich provides significant benefits in high-throughput scenarios. Enable DCP by adding\n`--decode-context-parallel-size 8`:\n\n```bash\nvllm serve moonshotai/Kimi-K2-Thinking \\\n --tensor-parallel-size 8 \\\n --decode-context-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k2 \\\n --reasoning-parser kimi_k2 \\\n --trust-remote-code\n```\n\n## Metrics (GSM8K)\n\n| Config | exact_match (flexible) | exact_match (strict) |\n|--------|-----------------------|----------------------|\n| TP8 | 0.9416 | 0.9409 |\n| TP8+DCP8 | 0.9386 | 0.9371 |\n\n## Benchmarking\n\nWe used the following script to benchmark `moonshotai/Kimi-K2-Thinking` on 8xH200:\n\n```bash\nvllm bench serve \\\n --model moonshotai/Kimi-K2-Thinking \\\n --dataset-name random \\\n --random-input 8000 \\\n --random-output 4000 \\\n --request-rate 100 \\\n --num-prompt 1000 \\\n --trust-remote-code\n```\n\n### DCP Gain Analysis\n\n| Metric | TP8 | TP8+DCP8 | Change | Improvement |\n|--------|-----|----------|--------|-------------|\n| Request throughput (req/s) | 1.25 | 1.57 | +0.32 | +25.6% |\n| Output token throughput (tok/s) | 485.78 | 695.13 | +209.35 | +43.1% |\n| Mean TTFT (s) | 271.2 | 227.8 | -43.4 | +16.0% |\n\nDCP multiplies the GPU KV cache size by `dcp_world_size`:\n- TP8 KV cache: `715,072` tokens\n- TP8+DCP8 KV cache: `5,721,088` tokens (8x)\n\nEnabling DCP delivers strong advantages (43% faster token generation, 26% higher\nthroughput) with minimal drawbacks. Read the\n[DCP doc](https://docs.vllm.ai/en/latest/serving/context_parallel_deployment.html#decode-context-parallel)\nand try it in your LLM workloads.\n\n## References\n\n- [Kimi-K2-Thinking on Hugging Face](https://huggingface.co/moonshotai/Kimi-K2-Thinking)\n- [vLLM Decode Context Parallel docs](https://docs.vllm.ai/en/latest/serving/context_parallel_deployment.html#decode-context-parallel)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/moonshotai/Kimi-K2-Thinking.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/moonshotai/Kimi-K2-Thinking.yaml", "overlays": [] } }