{ "hf_id": "dots-studio/dots3-note-prev", "meta": { "title": "dots3-note-prev", "slug": "dots3-note-prev", "provider": "Dots", "description": "Multimodal MoE model available in BF16 and native FP8, with hybrid DSA and SWA, 512K context, and MTP speculative decoding.", "date_added": "2026-08-13", "date_updated": "2026-09-09", "tasks": [ "multimodal", "text" ], "performance_headline": "Unified multimodal intelligence with strong agent capabilities", "hardware": { "h100": "verified", "b200": "unsupported", "gb200": "unsupported", "b300": "unsupported", "gb300": "unsupported", "mi300x": "unsupported", "mi325x": "unsupported", "mi355x": "unsupported" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tep", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve dots-studio/dots3-note-prev-fp8 \\\n --served-model-name dots3-note-prev \\\n --gpu-memory-utilization 0.9 \\\n --max-model-len 262144 \\\n --max-num-batched-tokens 8192 \\\n --block-size 64 \\\n --mm-processor-cache-gb 32 \\\n --enable-expert-parallel \\\n --tensor-parallel-size 8 \\\n --moe-backend deep_gemm", "argv": [ "vllm", "serve", "dots-studio/dots3-note-prev-fp8", "--served-model-name", "dots3-note-prev", "--gpu-memory-utilization", "0.9", "--max-model-len", "262144", "--max-num-batched-tokens", "8192", "--block-size", "64", "--mm-processor-cache-gb", "32", "--enable-expert-parallel", "--tensor-parallel-size", "8", "--moe-backend", "deep_gemm" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest dots-studio/dots3-note-prev-fp8 \\\n --served-model-name dots3-note-prev \\\n --gpu-memory-utilization 0.9 \\\n --max-model-len 262144 \\\n --max-num-batched-tokens 8192 \\\n --block-size 64 \\\n --mm-processor-cache-gb 32 \\\n --enable-expert-parallel \\\n --tensor-parallel-size 8 \\\n --moe-backend deep_gemm", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "dots-studio/dots3-note-prev-fp8", "--served-model-name", "dots3-note-prev", "--gpu-memory-utilization", "0.9", "--max-model-len", "262144", "--max-num-batched-tokens", "8192", "--block-size", "64", "--mm-processor-cache-gb", "32", "--enable-expert-parallel", "--tensor-parallel-size", "8", "--moe-backend", "deep_gemm" ], "strategy_spec": { "name": "single_node_tep", "deploy_type": "single_node", "display_name": "Tensor + Expert Parallel", "short_label": "TEP", "orientation": "balanced", "description": "Single-node TEP. TP splits dense layers and EP splits expert layers across local GPUs. TP must be set to GPU count to avoid OOM from replicated dense layers. For MoE models only.\n", "hardware_match": { "min_gpus": 2, "max_gpus": 8, "multi_node": false, "architecture": "moe" }, "vllm_args": [ "--enable-expert-parallel" ], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e ยท 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tp": "/dots-studio/dots3-note-prev/strategies/single_node_tp.json", "single_node_dep": "/dots-studio/dots3-note-prev/strategies/single_node_dep.json", "kv_store_centralized_mooncake": "/dots-studio/dots3-note-prev/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/dots-studio/dots3-note-prev/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/dots-studio/dots3-note-prev/hw/h200.json", "h100": "/dots-studio/dots3-note-prev/hw/h100.json", "mi350x": "/dots-studio/dots3-note-prev/hw/mi350x.json" } }, "model": { "model_id": "dots-studio/dots3-note-prev", "min_vllm_version": "0.28.0", "architecture": "moe", "parameter_count": "288B", "active_parameters": "17B", "context_length": 524288, "base_args": [ "--served-model-name", "dots3-note-prev", "--gpu-memory-utilization", "0.9", "--max-model-len", "262144", "--max-num-batched-tokens", "8192", "--block-size", "64", "--mm-processor-cache-gb", "32" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "reasoning": { "description": "Extract Qwen3-style traces into the OpenAI reasoning field.", "args": [ "--reasoning-parser", "qwen3" ] }, "tool_calling": { "description": "Parse dots3-note-prev's native XML tool-call format with automatic tool choice.", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "dots" ] }, "text_only": { "description": "Load only the language model for text workloads, skipping the vision and audio encoders.", "args": [ "--language-model-only" ] }, "spec_decoding": { "description": "Enable MTP speculative decoding with the checkpoint's prediction head.", "args": [ "--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":3}" ] } }, "opt_in_features": [ "reasoning", "tool_calling", "text_only", "spec_decoding" ], "variants": { "default": { "model_id": "dots-studio/dots3-note-prev-fp8", "precision": "fp8", "vram_minimum_gb": 359, "description": "Native block-scaled FP8 language-model weights with BF16 multimodal encoders; verified on 8x 80GB NVIDIA GPUs.", "hardware_overrides": { "nvidia": { "extra_args": [ "--moe-backend", "deep_gemm" ] } }, "json": "/dots-studio/dots3-note-prev-fp8.json" }, "bf16": { "model_id": "dots-studio/dots3-note-prev", "precision": "bf16", "vram_minimum_gb": 692, "description": "Full BF16 checkpoint using the Triton unquantized-MoE backend; its indexed weights total about 576.9 GB before runtime and KV-cache memory.", "extra_args": [ "--gpu-memory-utilization", "0.95" ], "hardware_overrides": { "nvidia": { "extra_args": [ "--moe-backend", "triton" ] } } } }, "hardware_overrides": {}, "guide": "## Overview\n\ndots3-note-prev is a native multimodal Mixture-of-Experts model supporting text, image,\nvideo, and audio inputs through one model architecture. It is published as a full\nBF16 checkpoint (`dots-studio/dots3-note-prev`) and a native FP8 checkpoint\n(`dots-studio/dots3-note-prev-fp8`). Its language backbone has 256 routed experts\n(top-8) plus a shared expert and mixes DSA and SWA layers. Both releases expose\nMTP speculative decoding.\n\nThe model configuration supports up to 524,288 tokens. The commands below use a\nvalidated 262,144-token serving limit and 8,192-token chunked-prefill budget so the\nremaining GPU memory can be used for KV cache. Increase `--max-model-len` only after\nchecking the KV-cache budget for the target workload.\n\n## Prerequisites\n\n- 8 NVIDIA GPUs with at least 80 GB each for the native FP8 checkpoint.\n- The BF16 checkpoint contains about 576.9 GB of indexed weights and needs additional\n memory for the runtime and KV cache; the recipe budgets 692 GB aggregate VRAM.\n- A recent CUDA environment. Native FP8 uses DeepGEMM, while unquantized BF16\n uses the Triton MoE backend.\n- vLLM 0.28.0 or newer, which includes native dots3-note-prev support.\n\n## Launch commands\n\nThe examples expose an OpenAI-compatible endpoint on all network interfaces. Remove\n`--host 0.0.0.0` if the service should only be reachable locally. The general\nstrategy examples use the FP8 checkpoint. BF16 needs a different MoE backend and\na larger memory budget, so use the explicit BF16 command below instead of only\nreplacing the model ID.\n\n### BF16 tensor parallel + expert parallel (TP=8, EP=8)\n\nThis command serves the full multimodal BF16 checkpoint. Triton is required because\nDeepGEMM does not support unquantized MoE weights. A 0.95 memory-utilization target\nleaves enough KV-cache capacity for the 262,144-token serving limit after profiling\nthe multimodal encoders. If the target hardware cannot reserve that much memory,\nlower `--max-model-len` or add `--language-model-only` for text workloads.\n\n```bash\nvllm serve dots-studio/dots3-note-prev \\\n --served-model-name dots3-note-prev \\\n --host 0.0.0.0 \\\n --tensor-parallel-size 8 \\\n --enable-expert-parallel \\\n --gpu-memory-utilization 0.95 \\\n --max-model-len 262144 \\\n --max-num-batched-tokens 8192 \\\n --block-size 64 \\\n --mm-processor-cache-gb 32 \\\n --moe-backend triton\n```\n\n### Tensor parallel (TP=8)\n\nTP shards both dense and expert weights across all eight GPUs. It is the simplest\ndeployment, but it does not enable expert parallelism.\n\n```bash\nvllm serve dots-studio/dots3-note-prev-fp8 \\\n --served-model-name dots3-note-prev \\\n --host 0.0.0.0 \\\n --tensor-parallel-size 8 \\\n --gpu-memory-utilization 0.9 \\\n --max-model-len 262144 \\\n --max-num-batched-tokens 8192 \\\n --block-size 64 \\\n --mm-processor-cache-gb 32 \\\n --moe-backend deep_gemm\n```\n\n### Tensor parallel + expert parallel (TP=8, EP=8)\n\nThis is the recommended single-node configuration. Dense layers use TP while MoE\nexperts are distributed across the eight expert-parallel ranks.\n\n```bash\nvllm serve dots-studio/dots3-note-prev-fp8 \\\n --served-model-name dots3-note-prev \\\n --host 0.0.0.0 \\\n --tensor-parallel-size 8 \\\n --enable-expert-parallel \\\n --gpu-memory-utilization 0.9 \\\n --max-model-len 262144 \\\n --max-num-batched-tokens 8192 \\\n --block-size 64 \\\n --mm-processor-cache-gb 32 \\\n --moe-backend deep_gemm\n```\n\n### Data parallel + expert parallel (DP=8, EP=8)\n\nDP+EP is intended for throughput-oriented serving. Each DP rank owns a replica of\nthe non-expert weights, while routed experts are distributed across all eight ranks.\nIt therefore needs more GPU memory than TP+EP.\n\n```bash\nvllm serve dots-studio/dots3-note-prev-fp8 \\\n --served-model-name dots3-note-prev \\\n --host 0.0.0.0 \\\n --data-parallel-size 8 \\\n --enable-expert-parallel \\\n --gpu-memory-utilization 0.9 \\\n --max-model-len 262144 \\\n --max-num-batched-tokens 8192 \\\n --block-size 64 \\\n --mm-processor-cache-gb 32 \\\n --moe-backend deep_gemm\n```\n\n### Language-model-only serving\n\nThe checkpoint keeps a single multimodal architecture and model type. For pure-text\nserving, add `--language-model-only`; a separate text checkpoint is not required.\nThe option can be combined with TP, TP+EP, or DP+EP.\n\n```bash\nvllm serve dots-studio/dots3-note-prev-fp8 \\\n --served-model-name dots3-note-prev \\\n --host 0.0.0.0 \\\n --tensor-parallel-size 8 \\\n --enable-expert-parallel \\\n --language-model-only \\\n --gpu-memory-utilization 0.9 \\\n --max-model-len 262144 \\\n --max-num-batched-tokens 8192 \\\n --block-size 64 \\\n --mm-processor-cache-gb 32 \\\n --moe-backend deep_gemm\n```\n\n### MTP speculative decoding\n\nAdd the speculative configuration to any of the deployments above. The current\ncheckpoints expose an MTP prediction head that vLLM reuses for drafting without\nloading a separate draft model. The example uses TP+EP with the native FP8 checkpoint.\n\n```bash\nvllm serve dots-studio/dots3-note-prev-fp8 \\\n --served-model-name dots3-note-prev \\\n --host 0.0.0.0 \\\n --tensor-parallel-size 8 \\\n --enable-expert-parallel \\\n --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' \\\n --gpu-memory-utilization 0.9 \\\n --max-model-len 262144 \\\n --max-num-batched-tokens 8192 \\\n --block-size 64 \\\n --mm-processor-cache-gb 32 \\\n --moe-backend deep_gemm\n```\n\n### Tool calling\n\ndots3-note-prev emits tool calls inside `` XML wrappers. Enable\nautomatic tool choice with the native `dots` parser. These options can be combined\nwith any parallel strategy, `--language-model-only`, and MTP speculative decoding.\n\n```bash\nvllm serve dots-studio/dots3-note-prev-fp8 \\\n --served-model-name dots3-note-prev \\\n --host 0.0.0.0 \\\n --tensor-parallel-size 8 \\\n --enable-expert-parallel \\\n --enable-auto-tool-choice \\\n --tool-call-parser dots \\\n --gpu-memory-utilization 0.9 \\\n --max-model-len 262144 \\\n --max-num-batched-tokens 8192 \\\n --block-size 64 \\\n --mm-processor-cache-gb 32 \\\n --moe-backend deep_gemm\n```\n\n## Client usage\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"dots3-note-prev\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Explain sparse attention briefly.\"}],\n \"temperature\": 0.7\n }'\n```\n\n## Tuning notes\n\n- `--max-num-batched-tokens 8192` is the chunked-prefill budget, not the maximum\n request length. Increase it only when prefill throughput is more important than\n inter-token latency and enough temporary workspace is available.\n- DP+EP replicates the non-expert portion on every DP rank. Prefer TP+EP when memory\n capacity or long-context KV cache is the priority.\n- `--language-model-only` skips multimodal encoder loading but does not change the\n checkpoint's model type or architecture.\n- Full multimodal BF16 serving at 262,144 tokens may not have enough KV-cache memory\n with `--gpu-memory-utilization 0.9`; the BF16 variant uses `0.95`. Lower\n `--max-model-len` instead when the extra memory cannot be reserved.\n- DeepGEMM is specific to the native FP8 checkpoint in this recipe. BF16 is\n unquantized and must use `--moe-backend triton`.\n- MTP changes decode scheduling and memory use. Benchmark it against non-speculative\n decoding with the same prompt/output distribution before production deployment.\n- `--tool-call-parser dots` supports the model's XML invoke/parameter format and its\n JSON fallback format, including multiple tool calls in one response.\n\n## References\n\n- [dots3-note-prev FP8 on Hugging Face](https://huggingface.co/dots-studio/dots3-note-prev-fp8)\n- [dots3-note-prev BF16 on Hugging Face](https://huggingface.co/dots-studio/dots3-note-prev)\n- [vLLM dots3-note-prev support PR](https://github.com/vllm-project/vllm/pull/51255)\n- [vLLM parallelism and scaling guide](https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/dots-studio/dots3-note-prev.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/dots-studio/dots3-note-prev.yaml", "overlays": [] } }