# MiniMaxAI/MiniMax-M3.1-Flash — vLLM serving recipe > MiniMax M3.1-Flash vision-language MoE (~428B total / 23B active) with MSA sparse attention throughout and a built-in DSpark draft; quantization-aware trained, MXFP8 dense layers and NVFP4 experts. Served TP4 on NVIDIA Blackwell or AMD MI355X, prefill/decode disaggregated as TP4 + TP4 on MI355X or DEP2 + DEP4 on GB200/GB300, and TP8 on NVIDIA Hopper. Configuration: hardware `b300` (8× B300) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (NVFP4) · features `tool_calling`, `reasoning`, `spec_decoding` - Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/MiniMaxAI/MiniMax-M3.1-Flash - This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/MiniMaxAI/MiniMax-M3.1-Flash.json?hardware=b300 The commands below were rendered for exactly this configuration and take precedence over any example in the guide. ## Other configurations Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line. - `hardware`: `b200`, `gb200`, `b300`, `gb300`, `mi355x` - `variant`: `default`, `mxfp4`, `fp8` - `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_dep`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=`) - `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding`, `text_only`, `prefill_dbo_sp`, `mono_decode` (default: `tool_calling`, `reasoning`, `spec_decoding`) - `preset`: a tuned, evidence-backed configuration (`baseline` opts out of a default one). Each sets its own hardware and strategy: - `gb300_agentx_agg` — AgentX TP4 (Dynamo) (gb300 · TP · Agentic (InferenceX AgentX) · opt-in): `hardware=gb300&strategy=single_node_tp&preset=gb300_agentx_agg` - `gb300_agentx_pd` — AgentX PD (Dynamo) (gb300 · PD · Agentic (InferenceX AgentX) · opt-in): `hardware=gb300&strategy=pd_cluster&preset=gb300_agentx_pd` - `b300_agentx_tp4` — AgentX TP4 (b300 · TP · Agentic (InferenceX AgentX) · opt-in): `hardware=b300&strategy=single_node_tp&preset=b300_agentx_tp4` - `b300_agentx_tp2` — AgentX TP2 + KV offload (b300 · TP · Agentic (InferenceX AgentX) · opt-in): `hardware=b300&strategy=single_node_tp&preset=b300_agentx_tp2` - `gb200_agentx_agg` — AgentX TP4 (Dynamo) (gb200 · TP · Agentic (InferenceX AgentX) · opt-in): `hardware=gb200&strategy=single_node_tp&preset=gb200_agentx_agg` - `gb200_agentx_pd` — AgentX PD (Dynamo) (gb200 · PD · Agentic (InferenceX AgentX) · opt-in): `hardware=gb200&strategy=pd_cluster&preset=gb200_agentx_pd` ## Install Requires vLLM >= 0.33.0 (not released yet — nightly wheel). Docker image: `vllm/vllm-openai:minimax-m31-flash` ## Serve ```bash export VLLM_FLOAT32_MATMUL_PRECISION=high vllm serve MiniMaxAI/MiniMax-M3.1-Flash \ --block-size 128 \ --disable-hybrid-kv-cache-manager \ --tensor-parallel-size 8 \ --kv-cache-dtype nvfp4 \ --attention-config '{"indexer_kv_dtype": "nvfp4", "use_prefill_query_quantization": true}' \ --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \ --tool-call-parser minimax_m3 \ --enable-auto-tool-choice \ --reasoning-parser minimax_m3 \ --speculative-config '{"method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_TS","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' ``` Docker equivalent: ```bash docker run --gpus all \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -e VLLM_FLOAT32_MATMUL_PRECISION=high \ vllm/vllm-openai:minimax-m31-flash MiniMaxAI/MiniMax-M3.1-Flash \ --block-size 128 \ --disable-hybrid-kv-cache-manager \ --tensor-parallel-size 8 \ --kv-cache-dtype nvfp4 \ --attention-config '{"indexer_kv_dtype": "nvfp4", "use_prefill_query_quantization": true}' \ --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \ --tool-call-parser minimax_m3 \ --enable-auto-tool-choice \ --reasoning-parser minimax_m3 \ --speculative-config '{"method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_TS","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' ``` ## Guide > Background reference. Commands in it are examples and may target another configuration — use the Serve section above. ### Overview [MiniMax-M3.1-Flash](https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash) is a vision-language MoE model from [MiniMax](https://www.minimax.io/) that keeps the MoE architecture of [MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3): about 428B total / 23B active parameters and a 1M-token context, with MSA sparse attention in every layer. - **Quantization-aware training** — the checkpoint stores MXFP8 dense layers and NVFP4 experts. - **Built-in DSpark draft** — the `dspark/` directory of the checkpoint; `--speculative-config` needs no draft model path. - **Native multimodal** — image and video inputs. ### Prerequisites - **AMD:** MI355X (gfx950), ROCm 7.2+, TP4. - **NVIDIA Blackwell:** GB200, GB300, B200 or B300, TP4; NVFP4 MoE runs natively on the FP4 tensor cores. - **NVIDIA Hopper:** H100 or H200, TP8 on a full node, with the FP8 variant: FP8 KV and indexer caches, NVFP4 MoE on Marlin. - **`--block-size 128` is mandatory** (MSA sparse/index cache). ### Launching the Server #### Single node (TP4, AMD MI355X) ```bash VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \ vllm serve MiniMaxAI/MiniMax-M3.1-Flash \ -tp 4 \ --reasoning-parser minimax_m3 \ --block-size 128 \ --kv-cache-dtype nvfp4 \ --attention-config '{"indexer_kv_dtype": "nvfp4", "indexer_decode_cp": true}' \ --quantization-config.moe mxfp8 \ --speculative-config '{"method":"dspark","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic"}' \ --disable-hybrid-kv-cache-manager ``` - `--kv-cache-dtype nvfp4` and `indexer_kv_dtype: nvfp4` store the KV cache and the sparse-attention indexer's K cache in NVFP4. - `indexer_decode_cp` splits the indexer's decode scoring across the TP ranks by context instead of by index head. - `--quantization-config.moe mxfp8` re-quantizes the NVFP4 experts to MXFP8 at load for the AITER MoE. #### DBO and SP prefill (TP4, AMD MI355X) Two opt-in switches make long prefill steps faster on the single-node MI355X engine (the **DBO / SP prefill** feature in the command builder): - **DBO** (dual-batch overlap): `--enable-dbo` splits a prefill step into two microbatches and runs one microbatch's all-reduce under the other's compute. `--dbo-decode-token-threshold 32768` keeps decode steps whole. DBO needs `VLLM_USE_V2_MODEL_RUNNER=1`: without it, `--enable-dbo` selects the V1 runner, which refuses the DSpark draft. - **SP** (sequence-parallel prefill): `VLLM_ROCM_MINIMAX_M3_SP_PREFILL=1` keeps the residual stream sharded by tokens on steps of 4096+ tokens, so each layer's two all-reduces become an all-to-all and a reduce-scatter. It keeps a full `o_proj` copy per GPU, about 3.1 GB less KV cache. - **SP + DBO**: set both, with `--dbo-prefill-token-threshold 16384`, so each DBO microbatch has 8192+ tokens and stays on the SP path. On the MXFP8 MoE, one 32K-token prefill takes 1035 ms plain, 655 ms with DBO and 484 ms with SP + DBO. The server log confirms SP with `MiniMax-M3 sequence-parallel prefill (from 4096 tokens, DBO microbatches from 8192): on`. ```bash VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \ VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ROCM_MINIMAX_M3_SP_PREFILL=1 \ vllm serve MiniMaxAI/MiniMax-M3.1-Flash \ -tp 4 \ --reasoning-parser minimax_m3 \ --block-size 128 \ --kv-cache-dtype nvfp4 \ --attention-config '{"indexer_kv_dtype": "nvfp4", "indexer_decode_cp": true}' \ --quantization-config.moe mxfp8 \ --speculative-config '{"method":"dspark","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic"}' \ --enable-dbo \ --dbo-prefill-token-threshold 16384 \ --dbo-decode-token-threshold 32768 \ --disable-hybrid-kv-cache-manager ``` #### Single node (TP4, NVIDIA GB200) ```bash VLLM_FLOAT32_MATMUL_PRECISION=high \ vllm serve MiniMaxAI/MiniMax-M3.1-Flash \ -tp 4 \ --reasoning-parser minimax_m3 \ --block-size 128 \ --kv-cache-dtype nvfp4 \ --attention-config '{"indexer_kv_dtype": "nvfp4", "use_prefill_query_quantization": true}' \ --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \ --speculative-config '{"method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_TS","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' \ --max-num-seqs 256 \ --disable-hybrid-kv-cache-manager ``` - Blackwell runs the NVFP4 experts natively, so no `--quantization-config.moe` override is needed. - `use_prefill_query_quantization` quantizes the prefill query on the MSA indexer path (Q8KV4). #### Single node (TP8, NVIDIA H100 / H200) ```bash vllm serve MiniMaxAI/MiniMax-M3.1-Flash \ --tensor-parallel-size 8 \ --reasoning-parser minimax_m3 \ --block-size 128 \ --kv-cache-dtype fp8 \ --attention-config '{"indexer_kv_dtype": "fp8"}' \ --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \ --max-model-len 262144 \ --max-num-seqs 80 \ --limit-mm-per-prompt '{"video": 0}' \ --gpu-memory-utilization 0.9 \ --no-enable-log-requests \ --disable-hybrid-kv-cache-manager \ --moe-backend marlin \ --enable-auto-tool-choice \ --tool-call-parser minimax_m3 ``` - Hopper has no FP4 tensor cores: the KV cache and the indexer K cache use FP8, and the NVFP4 experts run on the Marlin MoE kernel. - `--limit-mm-per-prompt '{"video": 0}'` turns off video inputs; images still work. - DSpark speculative decoding is not enabled on Hopper. #### Prefill/decode disaggregation (1P1D, TP4 + TP4, AMD MI355X) Prefill node: ```bash VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \ VLLM_KV_CACHE_LAYOUT=BLHNC VLLM_NIXL_SIDE_CHANNEL_HOST= \ vllm serve MiniMaxAI/MiniMax-M3.1-Flash \ -tp 4 \ --reasoning-parser minimax_m3 \ --block-size 128 \ --kv-cache-dtype nvfp4 \ --attention-config '{"indexer_kv_dtype": "nvfp4"}' \ --quantization-config.moe mxfp8 \ --speculative-config '{"method":"dspark","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic"}' \ --disable-hybrid-kv-cache-manager \ --max-num-batched-tokens 32768 \ --port 8100 \ --kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[ {"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_connector_extra_config":{"num_threads":4}}, {"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":601295421440,"cpu_hugepage_block_size":"1GB","lazy_offload":false}}]}}' ``` Decode node: ```bash VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \ VLLM_KV_CACHE_LAYOUT=BLHNC VLLM_NIXL_SIDE_CHANNEL_HOST= \ vllm serve MiniMaxAI/MiniMax-M3.1-Flash \ -tp 4 \ --reasoning-parser minimax_m3 \ --block-size 128 \ --kv-cache-dtype nvfp4 \ --attention-config '{"indexer_kv_dtype": "nvfp4", "indexer_decode_cp": true}' \ --quantization-config.moe mxfp8 \ --speculative-config '{"method":"dspark","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic"}' \ --disable-hybrid-kv-cache-manager \ --port 8200 \ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"num_threads":4}}' ``` - `` and `` are each node's own address, reachable from the other node (the NIXL side channel). - `VLLM_KV_CACHE_LAYOUT=BLHNC` goes on both nodes. It keeps all layers of a block contiguous, so NIXL reads one region per block instead of one per layer. - The prefill node never decodes, so it leaves `indexer_decode_cp` off. - The prefill node also offloads KV to host memory (`SimpleCPUOffloadConnector`): 560 GiB per rank in 1 GiB HugeTLB pages, so 2240 pages for the four ranks. Reserve them on the host before the start (and allow them in the container's hugetlb limit), or the server fails at startup with `Cannot map 560.00 GiB of 1024 MiB HugeTLB pages`: ```bash echo 2240 | sudo tee /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages ``` #### Prefill/decode disaggregation (1P1D, DEP2 + DEP4, NVIDIA GB200 / GB300) A DEP2 prefill on one 4-GPU tray feeds a DEP4 decode on a second tray over NIXL. Both roles run attention data parallel with the NVFP4 experts on the DeepGEMM MegaMoE kernel over expert parallelism, the V2 model runner, NUMA binding, and a seven-token DSpark draft on the FlashInfer PrimiTS (`FLASHINFER_TS`) backend. The settings were tuned on GB300; GB200 uses the same commands. Prefill node: ```bash VLLM_FLOAT32_MATMUL_PRECISION=high VLLM_USE_V2_MODEL_RUNNER=1 \ VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_NIXL_SIDE_CHANNEL_HOST= \ vllm serve MiniMaxAI/MiniMax-M3.1-Flash \ --data-parallel-size 2 \ --enable-expert-parallel \ --moe-backend deep_gemm_mega_moe \ --numa-bind \ --reasoning-parser minimax_m3 \ --block-size 128 \ --kv-cache-dtype nvfp4 \ --attention-config '{"indexer_kv_dtype": "nvfp4", "use_prefill_query_quantization": true}' \ --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \ --speculative-config '{"method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_TS","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' \ --max-num-seqs 64 \ --max-num-batched-tokens 32768 \ --long-prefill-token-threshold 8192 \ --gpu-memory-utilization 0.9 \ --enable-prefix-caching \ --disable-hybrid-kv-cache-manager \ --port 8100 \ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail","kv_connector_extra_config":{"engine_ttl":0}}' ``` Decode node: ```bash VLLM_FLOAT32_MATMUL_PRECISION=high VLLM_USE_V2_MODEL_RUNNER=1 \ VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_NIXL_SIDE_CHANNEL_HOST= \ vllm serve MiniMaxAI/MiniMax-M3.1-Flash \ --data-parallel-size 4 \ --enable-expert-parallel \ --moe-backend deep_gemm_mega_moe \ --numa-bind \ --reasoning-parser minimax_m3 \ --block-size 128 \ --kv-cache-dtype nvfp4 \ --attention-config '{"indexer_kv_dtype": "nvfp4", "use_prefill_query_quantization": true}' \ --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "max_cudagraph_capture_size": 224}' \ --speculative-config '{"method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_TS","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' \ --max-num-seqs 28 \ --max-num-batched-tokens 8192 \ --gpu-memory-utilization 0.95 \ --watermark 0.05 \ --no-enable-prefix-caching \ --disable-hybrid-kv-cache-manager \ --port 8200 \ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail","kv_connector_extra_config":{"engine_ttl":0}}' ``` - `VLLM_ENGINE_READY_TIMEOUT_S=3600`: with data parallel, each API server waits this long for its engines to start (default 600 s). A cold prefill start (JIT builds, CUDA graph capture, FlashInfer autotune) takes longer than 10 minutes. - `--max-num-seqs` is per DP rank. The decode CUDA graphs cover 28 requests of 1 + 7 tokens (224). - `--watermark 0.05` keeps 5% of the decode KV cache free when admitting requests, so extra requests wait in the queue instead of preempting running ones. On GB300 (about 4.5M KV tokens per decode rank) at a 200K-token mean context, expect about 19-21 running requests per rank. - Keep the decode `--max-num-batched-tokens` at 8192: a preempted request recomputes its context in chunks of that size, and every DP rank runs eager steps until it finishes. - `engine_ttl: 0` turns off NIXL's cleanup of idle remote engines; with the default one-hour TTL, a decode crashed an hour after its prefill went away. - Prefix caching stays on the prefill node only. For multi-turn traffic, the prefill node can also offload KV to host memory, 180 GiB per rank: `--kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_connector_extra_config":{"engine_ttl":0}},{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":193273528320,"lazy_offload":true}}]}}'`. For a quick test, put the toy NIXL proxy from the vLLM repository in front of the two nodes and send requests to port 8000: ```bash python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py --port 8000 \ --prefiller-host --prefiller-port 8100 \ --decoder-host --decoder-port 8200 ``` ### Client Usage Recommended sampling parameters (from the model card): `temperature = 1.0`, `top_p = 0.95`. Set the reasoning effort per request with `chat_template_kwargs: {"reasoning_effort": ""}`: `low`, `medium`, `high`, `xhigh` or `max`. ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "MiniMaxAI/MiniMax-M3.1-Flash", "temperature": 1.0, "top_p": 0.95, "messages": [ {"role": "user", "content": "Explain MSA sparse attention in 3 bullets."} ] }' ``` ### Benchmarking ```bash vllm bench serve \ --backend vllm \ --model MiniMaxAI/MiniMax-M3.1-Flash \ --endpoint /v1/completions \ --dataset-name random \ --random-input 8192 \ --random-output 1024 \ --max-concurrency 32 \ --num-prompts 320 ``` ### Troubleshooting - **`--block-size` mismatch.** MSA's sparse block size is 128; the vLLM KV cache block size must match (otherwise `No common block size for 16`). - **Parsers.** `--tool-call-parser` and `--reasoning-parser` both use `minimax_m3`. - **HugeTLB pages (PD prefill node).** See the AMD PD section above. ### References - [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash) - [MiniMax-M3 recipe](https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3) - [MiniMax](https://www.minimax.io/)