{ "hf_id": "deepseek-ai/DeepSeek-V3.2-Exp", "meta": { "title": "DeepSeek-V3.2-Exp", "slug": "deepseek-v3.2-exp", "provider": "DeepSeek", "description": "Experimental DeepSeek-V3.2 preview with sparse attention (MQA-like logits) and FP8 KV cache; architecture matches DeepSeek-V3.1 except for the sparse attention mechanism.", "date_added": "2025-09-29", "date_updated": "2026-09-14", "tasks": [ "text" ], "performance_headline": "Sparse attention MoE with FP8 KV cache and strong GSM8K score (~0.96)", "related_recipes": [ "deepseek-ai/DeepSeek-V3.1", "deepseek-ai/DeepSeek-V3.2" ], "hardware": { "h200": "verified", "mi300x": "verified", "mi325x": "verified", "mi355x": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": { "VLLM_USE_RUST_FRONTEND": "1" }, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve deepseek-ai/DeepSeek-V3.2-Exp \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --tokenizer-mode deepseek_v32 \\\n --tool-call-parser deepseek_v32 \\\n --enable-auto-tool-choice \\\n --reasoning-parser deepseek_v3", "argv": [ "vllm", "serve", "deepseek-ai/DeepSeek-V3.2-Exp", "--trust-remote-code", "--tensor-parallel-size", "8", "--tokenizer-mode", "deepseek_v32", "--tool-call-parser", "deepseek_v32", "--enable-auto-tool-choice", "--reasoning-parser", "deepseek_v3" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:latest deepseek-ai/DeepSeek-V3.2-Exp \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --tokenizer-mode deepseek_v32 \\\n --tool-call-parser deepseek_v32 \\\n --enable-auto-tool-choice \\\n --reasoning-parser deepseek_v3", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai:latest", "deepseek-ai/DeepSeek-V3.2-Exp", "--trust-remote-code", "--tensor-parallel-size", "8", "--tokenizer-mode", "deepseek_v32", "--tool-call-parser", "deepseek_v32", "--enable-auto-tool-choice", "--reasoning-parser", "deepseek_v3" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tep": "/deepseek-ai/DeepSeek-V3.2-Exp/strategies/single_node_tep.json", "single_node_dep": "/deepseek-ai/DeepSeek-V3.2-Exp/strategies/single_node_dep.json", "multi_node_tp": "/deepseek-ai/DeepSeek-V3.2-Exp/strategies/multi_node_tp.json", "multi_node_tp_pp": "/deepseek-ai/DeepSeek-V3.2-Exp/strategies/multi_node_tp_pp.json", "multi_node_dep": "/deepseek-ai/DeepSeek-V3.2-Exp/strategies/multi_node_dep.json", "multi_node_tep": "/deepseek-ai/DeepSeek-V3.2-Exp/strategies/multi_node_tep.json", "kv_store_centralized_mooncake": "/deepseek-ai/DeepSeek-V3.2-Exp/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/deepseek-ai/DeepSeek-V3.2-Exp/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/deepseek-ai/DeepSeek-V3.2-Exp/hw/h200.json", "h100": "/deepseek-ai/DeepSeek-V3.2-Exp/hw/h100.json", "b200": "/deepseek-ai/DeepSeek-V3.2-Exp/hw/b200.json", "gb200": "/deepseek-ai/DeepSeek-V3.2-Exp/hw/gb200.json", "b300": "/deepseek-ai/DeepSeek-V3.2-Exp/hw/b300.json", "gb300": "/deepseek-ai/DeepSeek-V3.2-Exp/hw/gb300.json", "mi300x": "/deepseek-ai/DeepSeek-V3.2-Exp/hw/mi300x.json", "mi325x": "/deepseek-ai/DeepSeek-V3.2-Exp/hw/mi325x.json", "mi350x": "/deepseek-ai/DeepSeek-V3.2-Exp/hw/mi350x.json", "mi355x": "/deepseek-ai/DeepSeek-V3.2-Exp/hw/mi355x.json" } }, "model": { "model_id": "deepseek-ai/DeepSeek-V3.2-Exp", "min_vllm_version": "0.12.0", "default_frontend": "rust", "architecture": "moe", "parameter_count": "671B", "active_parameters": "37B", "context_length": 163840, "supports_dcp": true, "base_args": [ "--trust-remote-code" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" }, "extras": [ { "note": "DeepGEMM pinned build for MQA logits (FP8 MoE kernels)", "command": "uv pip install git+https://github.com/deepseek-ai/DeepGEMM.git@v2.1.1.post3 --no-build-isolation" } ] } }, "features": { "tool_calling": { "description": "Enable tool calling with DeepSeek V3.2 chat template support.", "args": [ "--tokenizer-mode", "deepseek_v32", "--tool-call-parser", "deepseek_v32", "--enable-auto-tool-choice" ] }, "reasoning": { "description": "Dynamic thinking mode via chat_template_kwargs, same as DeepSeek-V3.1.", "args": [ "--reasoning-parser", "deepseek_v3" ] } }, "opt_in_features": [], "variants": { "default": { "precision": "fp8", "vram_minimum_gb": 805, "description": "Native FP8 weights on 8xH200 (or H20, or 8xB200) with FP8 KV cache default" } }, "hardware_overrides": { "amd": { "extra_args": [ "--block-size", "1", "--kv-cache-dtype", "bfloat16", "--max-num-batched-tokens", "32768" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1", "VLLM_ROCM_USE_AITER_MOE": "1", "SAFETENSORS_FAST_GPU": "1" } } }, "guide": "## Overview\n\nDeepSeek-V3.2-Exp is a sparse-attention MoE preview. Its main architecture is similar to\nDeepSeek-V3.1, with a sparse attention mechanism. This guide covers deployment on NVIDIA\nHopper/Blackwell GPUs and AMD MI300X / MI325X / MI355X GPUs.\n\n## Prerequisites\n\n- **Hardware**: 8x H200/H20/B200 (CUDA) or 8x MI300X / MI325X / MI355X (ROCm)\n- **vLLM**: Current stable release\n- **DeepGEMM** (CUDA only): Required for MQA logits computation (and optionally for MoE)\n\n### CUDA\n```bash\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto\nuv pip install git+https://github.com/deepseek-ai/DeepGEMM.git@v2.1.1.post3 --no-build-isolation\n```\n\n### ROCm (MI300X, MI325X, MI355X)\nRequires Python 3.12, ROCm 7.2.1, and glibc >= 2.35. If your environment does not\nmeet these requirements, use the Docker-based setup described in the\n[documentation](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/#pre-built-images).\n```bash\nuv venv --python 3.12\nsource .venv/bin/activate\nuv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/\n```\n\nNote: DeepGEMM is used both for MoE and for MQA logits computation. It is required for\nMQA logits on CUDA. To disable the MoE path only, set `VLLM_USE_DEEP_GEMM=0`. Some users\nreport better performance with `VLLM_USE_DEEP_GEMM=0` (e.g. on H20), and this also skips\nthe long warmup.\n\n## Launching DeepSeek-V3.2-Exp\n\n### 8xH200 / 8xH20 (CUDA)\n\nUsing the recommended EP/DP mode:\n```bash\nvllm serve deepseek-ai/DeepSeek-V3.2-Exp -dp 8 --enable-expert-parallel\n```\n\nUsing tensor parallel:\n```bash\nvllm serve deepseek-ai/DeepSeek-V3.2-Exp -tp 8\n```\n\n### 8xB200 (CUDA)\n\nSame as 8xH200 / 8xH20.\n\n### 8xMI300X / MI325X / MI355X (ROCm)\n\n```bash\nexport SAFETENSORS_FAST_GPU=1\nexport VLLM_ROCM_USE_AITER=1\nexport VLLM_ROCM_USE_AITER_MOE=1\n\nvllm serve deepseek-ai/DeepSeek-V3.2-Exp \\\n --tensor-parallel-size 8 \\\n --max-num-batched-tokens 32768 \\\n --trust-remote-code \\\n --kv-cache-dtype bfloat16 \\\n --block-size 1\n```\n\n## Accuracy Benchmarking\n\n```bash\nlm-eval --model local-completions --tasks gsm8k \\\n --model_args model=deepseek-ai/DeepSeek-V3.2-Exp,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=100,max_retries=3,tokenized_requests=False\n```\n\nReported GSM8K score: `0.9591` (5-shot) and `0.9538` (20-shot).\n\n## Performance Tips\n\n1. The kernels are mainly optimized for TP=1, so it is recommended to run this model\n under EP/DP mode (e.g. DP=8, EP=8, TP=1). If you hit any errors or hangs, try tensor\n parallel instead. Simple TP works and is more robust, but the performance is not optimal.\n2. The default config uses a custom `fp8` KV cache. You can also use `bfloat16` KV cache\n by specifying `kv_cache_dtype=bfloat16`. FP8 allows more tokens to be cached but incurs\n quantization/dequantization overhead. Use `bfloat16` for short requests and `fp8` for\n long requests.\n\n## Troubleshooting\n\n- **`CUDA error (flashmla-src/csrc/smxx/mla_combine.cu:201): invalid configuration argument`**:\n This may be caused by too large a batch size. Try `--max-num-seqs 256` or smaller\n (default is 1024).\n- For thinking-mode toggling, refer to the DeepSeek-V3.1 recipe (`deepseek-ai/DeepSeek-V3.1`).\n\n## References\n\n- [DeepSeek-V3.2-Exp on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp)\n- [End-to-end tutorial (Jupyter Notebook)](https://github.com/vllm-project/recipes/blob/main/DeepSeek/DeepSeek_v3_2_vLLM_getting_started_guide.ipynb)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/deepseek-ai/DeepSeek-V3.2-Exp.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/deepseek-ai/DeepSeek-V3.2-Exp.yaml", "overlays": [] } }