{ "hf_id": "moonshotai/Kimi-Linear-48B-A3B-Instruct", "meta": { "title": "Kimi-Linear-48B-A3B-Instruct", "slug": "kimi-linear-48b-a3b-instruct", "provider": "Moonshot AI", "description": "Kimi-Linear is a 48B-parameter instruction-tuned MoE model (~3B activated) with a linear-attention variant supporting very long context (1M tokens).", "date_added": "2025-10-31", "date_updated": "2026-04-17", "tasks": [ "text" ], "performance_headline": "Linear-attention MoE with 1M-token context on a single node", "related_recipes": [], "hardware": { "h200": "verified", "mi300x": "verified", "mi325x": "verified", "mi355x": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve moonshotai/Kimi-Linear-48B-A3B-Instruct \\\n --trust-remote-code \\\n --tensor-parallel-size 1", "argv": [ "vllm", "serve", "moonshotai/Kimi-Linear-48B-A3B-Instruct", "--trust-remote-code", "--tensor-parallel-size", "1" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest moonshotai/Kimi-Linear-48B-A3B-Instruct \\\n --trust-remote-code \\\n --tensor-parallel-size 1", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "moonshotai/Kimi-Linear-48B-A3B-Instruct", "--trust-remote-code", "--tensor-parallel-size", "1" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "multi_node_tp": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/strategies/multi_node_tp.json", "multi_node_tp_pp": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/strategies/multi_node_tp_pp.json", "multi_node_tep": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/strategies/multi_node_tep.json", "multi_node_dep": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/strategies/multi_node_dep.json", "kv_store_centralized_mooncake": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/hw/h200.json", "h100": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/hw/h100.json", "b200": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/hw/b200.json", "gb200": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/hw/gb200.json", "b300": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/hw/b300.json", "gb300": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/hw/gb300.json", "mi300x": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/hw/mi300x.json", "mi325x": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/hw/mi325x.json", "mi350x": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/hw/mi350x.json", "mi355x": "/moonshotai/Kimi-Linear-48B-A3B-Instruct/hw/mi355x.json" } }, "model": { "model_id": "moonshotai/Kimi-Linear-48B-A3B-Instruct", "min_vllm_version": "0.11.2", "architecture": "moe", "parameter_count": "48B", "active_parameters": "3B", "context_length": 1048576, "base_args": [ "--trust-remote-code" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" }, "extras": [ { "note": "Pin vllm==0.11.2 — 0.12.0 has a known Kimi-Linear regression", "command": "uv pip install vllm==0.11.2 --torch-backend auto" } ] } }, "features": {}, "opt_in_features": [], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 115, "description": "Full precision BF16 on 4 or 8 GPUs (single node)" } }, "hardware_overrides": { "amd": { "extra_args": [ "--no-enable-prefix-caching" ], "extra_env": { "SAFETENSORS_FAST_GPU": "1" } } }, "guide": "## Overview\n\nKimi-Linear is Moonshot AI's 48B-parameter instruction-tuned MoE model (`A3B` indicates\n~3B active parameters per token) featuring a linear-attention variant that enables very\nlong context windows (up to 1,048,576 tokens).\n\n## Prerequisites\n\n- **Hardware**: 4 or 8 GPUs on a single node\n- **vLLM**: `0.11.2` recommended. Avoid vLLM `0.12.0`, which has a known bug\n `MLAModules.__init__() missing 1 required positional argument: 'indexer_rotary_emb'`\n that affects Kimi-Linear.\n\n```bash\nuv venv\nsource .venv/bin/activate\n# Install a stable version (avoid 0.12.0)\nuv pip install vllm==0.11.2 --torch-backend auto\n```\n\n### AMD ROCm\n\nThe ROCm wheel requires Python 3.12, ROCm 7.0+, and glibc >= 2.35. Use the\nROCm Docker image when your host environment does not meet those requirements.\n\n```bash\nuv venv --python 3.12\nsource .venv/bin/activate\nuv pip install vllm==0.11.2 --extra-index-url https://wheels.vllm.ai/rocm/\n```\n\n## Running Kimi-Linear\n\nThe following snippets assume 4 or 8 GPUs on a single node.\n\n### 4-GPU Tensor Parallel\n\n```bash\nvllm serve moonshotai/Kimi-Linear-48B-A3B-Instruct \\\n --port 8000 \\\n --tensor-parallel-size 4 \\\n --max-model-len 1048576 \\\n --trust-remote-code\n```\n\n### 8-GPU Tensor Parallel\n\n```bash\nvllm serve moonshotai/Kimi-Linear-48B-A3B-Instruct \\\n --port 8000 \\\n --tensor-parallel-size 8 \\\n --max-model-len 1048576 \\\n --trust-remote-code\n```\n\n### ROCm on 8x MI300X / MI325X / MI355X\n\n```bash\nexport SAFETENSORS_FAST_GPU=1\nvllm serve moonshotai/Kimi-Linear-48B-A3B-Instruct \\\n --tensor-parallel-size 8 \\\n --max-model-len auto \\\n --trust-remote-code\n```\n\nIf you see OOM, reduce `--max-model-len` (e.g. 65536) or increase\n`--gpu-memory-utilization` (<= 0.95).\n\n## Client Usage\n\nOnce the server is up, test it with:\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\"model\":\"moonshotai/Kimi-Linear-48B-A3B-Instruct\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello!\"}]}'\n```\n\n## Troubleshooting\n\n- **`MLAModules.__init__() missing 1 required positional argument: 'indexer_rotary_emb'`**:\n Known bug in vLLM 0.12.0 affecting Kimi-Linear. Pin to `vllm==0.11.2` instead.\n- **OOM**: Reduce `--max-model-len` or increase `--gpu-memory-utilization` up to 0.95.\n\n## References\n\n- [Kimi-Linear-48B-A3B-Instruct on Hugging Face](https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/moonshotai/Kimi-Linear-48B-A3B-Instruct.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/moonshotai/Kimi-Linear-48B-A3B-Instruct.yaml", "overlays": [] } }