{
"hf_id": "deepseek-ai/DeepSeek-V3.1",
"meta": {
"title": "DeepSeek-V3.1",
"slug": "deepseek-v3.1",
"provider": "DeepSeek",
"description": "DeepSeek-V3.1 is a hybrid MoE model that supports dynamic switching between thinking and non-thinking modes, with tool calling and function execution.",
"date_added": "2025-08-21",
"date_updated": "2026-09-14",
"tasks": [
"text"
],
"performance_headline": "Hybrid thinking / non-thinking MoE with native FP8 and tool calling",
"related_recipes": [
"deepseek-ai/DeepSeek-V3",
"deepseek-ai/DeepSeek-V3.2-Exp"
],
"hardware": {
"h200": "verified",
"mi300x": "verified",
"mi325x": "verified",
"mi355x": "verified"
}
},
"recommended_command": {
"hardware": "h200",
"strategy": "single_node_tp",
"variant": "default",
"node_count": 1,
"deploy_type": "single_node",
"env": {
"VLLM_USE_RUST_FRONTEND": "1"
},
"docker_image": "vllm/vllm-openai:latest",
"command": "vllm serve deepseek-ai/DeepSeek-V3.1 \\\n --trust-remote-code \\\n --enable-expert-parallel \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser deepseek_v31 \\\n --chat-template examples/tool_chat_template_deepseekv31.jinja \\\n --reasoning-parser deepseek_v3",
"argv": [
"vllm",
"serve",
"deepseek-ai/DeepSeek-V3.1",
"--trust-remote-code",
"--enable-expert-parallel",
"--tensor-parallel-size",
"8",
"--enable-auto-tool-choice",
"--tool-call-parser",
"deepseek_v31",
"--chat-template",
"examples/tool_chat_template_deepseekv31.jinja",
"--reasoning-parser",
"deepseek_v3"
],
"docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:latest deepseek-ai/DeepSeek-V3.1 \\\n --trust-remote-code \\\n --enable-expert-parallel \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser deepseek_v31 \\\n --chat-template examples/tool_chat_template_deepseekv31.jinja \\\n --reasoning-parser deepseek_v3",
"docker_argv": [
"docker",
"run",
"--gpus",
"all",
"--privileged",
"--ipc=host",
"-p",
"8000:8000",
"-v",
"~/.cache/huggingface:/root/.cache/huggingface",
"-e",
"VLLM_USE_RUST_FRONTEND=1",
"vllm/vllm-openai:latest",
"deepseek-ai/DeepSeek-V3.1",
"--trust-remote-code",
"--enable-expert-parallel",
"--tensor-parallel-size",
"8",
"--enable-auto-tool-choice",
"--tool-call-parser",
"deepseek_v31",
"--chat-template",
"examples/tool_chat_template_deepseekv31.jinja",
"--reasoning-parser",
"deepseek_v3"
],
"strategy_spec": {
"name": "single_node_tp",
"deploy_type": "single_node",
"display_name": "Tensor Parallel",
"short_label": "TP",
"orientation": "latency",
"description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n",
"hardware_match": {
"min_gpus": 1,
"max_gpus": 8,
"multi_node": false
},
"vllm_args": [],
"parallel_flag": "--tensor-parallel-size"
},
"hardware_profile": {
"brand": "NVIDIA",
"generation": "hopper",
"compute_arch": "sm90",
"display_name": "H200",
"description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node",
"gpu_count": 8,
"vram_gb": 1128,
"multi_node": false
},
"alternatives": {
"single_node_tep": "/deepseek-ai/DeepSeek-V3.1/strategies/single_node_tep.json",
"single_node_dep": "/deepseek-ai/DeepSeek-V3.1/strategies/single_node_dep.json",
"multi_node_tp": "/deepseek-ai/DeepSeek-V3.1/strategies/multi_node_tp.json",
"multi_node_tp_pp": "/deepseek-ai/DeepSeek-V3.1/strategies/multi_node_tp_pp.json",
"multi_node_dep": "/deepseek-ai/DeepSeek-V3.1/strategies/multi_node_dep.json",
"multi_node_tep": "/deepseek-ai/DeepSeek-V3.1/strategies/multi_node_tep.json",
"pd_cluster": "/deepseek-ai/DeepSeek-V3.1/strategies/pd_cluster.json",
"kv_store_centralized_mooncake": "/deepseek-ai/DeepSeek-V3.1/strategies/kv_store_centralized_mooncake.json",
"kv_store_distributed_mooncake": "/deepseek-ai/DeepSeek-V3.1/strategies/kv_store_distributed_mooncake.json"
},
"by_hardware": {
"h200": "/deepseek-ai/DeepSeek-V3.1/hw/h200.json",
"h100": "/deepseek-ai/DeepSeek-V3.1/hw/h100.json",
"b200": "/deepseek-ai/DeepSeek-V3.1/hw/b200.json",
"gb200": "/deepseek-ai/DeepSeek-V3.1/hw/gb200.json",
"b300": "/deepseek-ai/DeepSeek-V3.1/hw/b300.json",
"gb300": "/deepseek-ai/DeepSeek-V3.1/hw/gb300.json",
"mi300x": "/deepseek-ai/DeepSeek-V3.1/hw/mi300x.json",
"mi325x": "/deepseek-ai/DeepSeek-V3.1/hw/mi325x.json",
"mi350x": "/deepseek-ai/DeepSeek-V3.1/hw/mi350x.json",
"mi355x": "/deepseek-ai/DeepSeek-V3.1/hw/mi355x.json"
}
},
"model": {
"model_id": "deepseek-ai/DeepSeek-V3.1",
"min_vllm_version": "0.12.0",
"default_frontend": "rust",
"architecture": "moe",
"parameter_count": "671B",
"active_parameters": "37B",
"context_length": 163840,
"supports_dcp": true,
"base_args": [
"--trust-remote-code",
"--enable-expert-parallel"
],
"base_env": {},
"install": {
"pip": {
"command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto"
},
"docker": {
"command": "docker pull vllm/vllm-openai:latest"
}
}
},
"features": {
"tool_calling": {
"description": "Enable DeepSeek-V3.1 tool calling with the deepseek_v31 tool-call parser.",
"args": [
"--enable-auto-tool-choice",
"--tool-call-parser",
"deepseek_v31",
"--chat-template",
"examples/tool_chat_template_deepseekv31.jinja"
]
},
"reasoning": {
"description": "Dynamic thinking mode via chat_template_kwargs={'thinking': true|false}. No separate parser flag is required; the chat template emits ... content.",
"args": [
"--reasoning-parser",
"deepseek_v3"
]
}
},
"opt_in_features": [],
"variants": {
"default": {
"precision": "fp8",
"vram_minimum_gb": 805,
"description": "Native FP8 weights on 8xH200 (or H20) with 141GB per GPU"
},
"nvfp4": {
"model_id": "nvidia/DeepSeek-V3.1-NVFP4",
"precision": "nvfp4",
"vram_minimum_gb": 403,
"description": "NVIDIA NVFP4 quantized weights for Blackwell GPUs",
"extra_args": [
"--kv-cache-dtype",
"fp8"
],
"json": "/nvidia/DeepSeek-V3.1-NVFP4.json"
}
},
"hardware_overrides": {
"amd": {
"extra_args": [],
"extra_env": {
"VLLM_ROCM_USE_AITER": "1",
"VLLM_ROCM_USE_AITER_MOE": "1",
"SAFETENSORS_FAST_GPU": "1"
}
}
},
"guide": "## Overview\n\nDeepSeek-V3.1 is a hybrid MoE model that supports both thinking and non-thinking modes.\nYou can dynamically switch between the two modes from the client by passing\n`extra_body={\"chat_template_kwargs\": {\"thinking\": True|False}}`.\n\n## Prerequisites\n\n- **Hardware**: 8x H200/H20 (CUDA) or 8x MI300X / MI325X / MI355X (ROCm)\n- **vLLM**: Current stable release\n\n### CUDA\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto\n```\n\n### ROCm (MI300X, MI325X, MI355X)\nRequires Python 3.12, ROCm 7.2.1, and glibc >= 2.35.\n```bash\nuv venv --python 3.12\nsource .venv/bin/activate\nuv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/\n```\n\n## Launching DeepSeek-V3.1\n\n### 8xH200 / 8xH20 (CUDA)\n\n```bash\nvllm serve deepseek-ai/DeepSeek-V3.1 \\\n --enable-expert-parallel \\\n --tensor-parallel-size 8 \\\n --served-model-name ds31\n```\n\n### 8xMI300X / MI325X / MI355X (ROCm)\n\n```bash\nexport SAFETENSORS_FAST_GPU=1\nexport VLLM_ROCM_USE_AITER=1\nexport VLLM_ROCM_USE_AITER_MOE=1\n\nvllm serve deepseek-ai/DeepSeek-V3.1 \\\n --tensor-parallel-size 8 \\\n --enable-expert-parallel\n```\n\n### Function calling\n\nvLLM supports user-defined tool calling for DeepSeek-V3.1. Add these flags when launching\nthe server. The example chat template ships in the official container and can also be\ndownloaded from the vLLM repo:\n[`tool_chat_template_deepseekv31.jinja`](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_deepseekv31.jinja).\n\n```bash\nvllm serve ... \\\n --enable-auto-tool-choice \\\n --tool-call-parser deepseek_v31 \\\n --chat-template examples/tool_chat_template_deepseekv31.jinja\n```\n\n## Client Usage\n\n### OpenAI Python SDK\n\nControl thinking mode via `extra_body={\"chat_template_kwargs\": {\"thinking\": False}}`\n(or `True` to enable thinking).\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\nmodel = client.models.list().data[0].id\n\nmessages = [\n {\"role\": \"system\", \"content\": \"You are a helpful assistant\"},\n {\"role\": \"user\", \"content\": \"Who are you?\"},\n {\"role\": \"assistant\", \"content\": \"HmmI am DeepSeek\"},\n {\"role\": \"user\", \"content\": \"9.11 and 9.8, which is greater?\"},\n]\nresponse = client.chat.completions.create(\n model=model,\n messages=messages,\n extra_body={\"chat_template_kwargs\": {\"thinking\": False}},\n)\nprint(response.choices[0].message.content)\n```\n\nWhen `thinking=True`, output includes a `` segment delimiting chain-of-thought;\nwhen `thinking=False`, the model produces a direct answer without the thinking segment.\n\n### curl\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"ds31\",\n \"messages\": [\n {\"role\": \"user\", \"content\": \"9.11 and 9.8, which is greater?\"}\n ],\n \"chat_template_kwargs\": {\"thinking\": true}\n }'\n```\n\n## References\n\n- [DeepSeek-V3.1 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3.1)\n- [vLLM tool chat template for DeepSeek-V3.1](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_deepseekv31.jinja)\n",
"source": {
"source": "vllm",
"repo": "vllm-project/vllm",
"ref": "main",
"sha": "389ffebe555705c85337025d5df108d0384d365e",
"path": "recipes/models/deepseek-ai/DeepSeek-V3.1.yaml",
"edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/deepseek-ai/DeepSeek-V3.1.yaml",
"overlays": []
}
}