{ "hf_id": "meta-models/Muse-Spark-Open-816B-A42B", "meta": { "title": "Muse Spark Open", "slug": "muse-spark-open-816b-a42b", "provider": "Muse (Meta)", "description": "Sparse MoE reasoning model with 816B total / 42B active parameters and a 1M-token context, built for long-horizon agentic coding. Native image and video perception, channel-scoped reasoning and tool calls via the `muse` parsers, and an optional DSpark draft model for speculative decoding.", "date_added": "2026-10-15", "date_updated": "2026-10-16", "tasks": [ "text", "multimodal" ], "performance_headline": "MXFP4 and BF16 on one 8-GPU node: TP + expert parallel with DSpark for interactive serving, TP + expert parallel alone for peak MXFP4 throughput", "related_recipes": [ "meta-models/Muse-Glimmer-30B" ], "default_hardware": "mi355x", "default_variant": { "amd": "mxfp4" }, "hardware": { "h200": "verified", "gb200": "verified", "gb300": "verified", "mi350x": "verified", "mi355x": "verified" } }, "recommended_command": { "hardware": "b200", "strategy": "single_node_tep", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": { "VLLM_USE_RUST_FRONTEND": "1" }, "docker_image": "vllm/vllm-openai:muse-spark-open", "command": "vllm serve meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \\\n --enable-expert-parallel \\\n --tensor-parallel-size 8 \\\n --reasoning-parser muse \\\n --enable-auto-tool-choice \\\n --tool-call-parser muse \\\n --speculative-config '{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":true}'", "argv": [ "vllm", "serve", "meta-models/Muse-Spark-Open-816B-A42B-NVFP4", "--enable-expert-parallel", "--tensor-parallel-size", "8", "--reasoning-parser", "muse", "--enable-auto-tool-choice", "--tool-call-parser", "muse", "--speculative-config", "{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":true}" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:muse-spark-open meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \\\n --enable-expert-parallel \\\n --tensor-parallel-size 8 \\\n --reasoning-parser muse \\\n --enable-auto-tool-choice \\\n --tool-call-parser muse \\\n --speculative-config '{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":true}'", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai:muse-spark-open", "meta-models/Muse-Spark-Open-816B-A42B-NVFP4", "--enable-expert-parallel", "--tensor-parallel-size", "8", "--reasoning-parser", "muse", "--enable-auto-tool-choice", "--tool-call-parser", "muse", "--speculative-config", "{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":true}" ], "strategy_spec": { "name": "single_node_tep", "deploy_type": "single_node", "display_name": "Tensor + Expert Parallel", "short_label": "TEP", "orientation": "balanced", "description": "Single-node TEP. TP splits dense layers and EP splits expert layers across local GPUs. TP must be set to GPU count to avoid OOM from replicated dense layers. For MoE models only.\n", "hardware_match": { "min_gpus": 2, "max_gpus": 8, "multi_node": false, "architecture": "moe" }, "vllm_args": [ "--enable-expert-parallel" ], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "blackwell", "compute_arch": "sm100", "display_name": "B200", "description": "NVIDIA B200 SXM 180 GB HBM3e · 8-GPU HGX B200 node", "gpu_count": 8, "vram_gb": 1440, "multi_node": false }, "alternatives": { "single_node_tp": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/single_node_tp.json", "single_node_dep": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/single_node_dep.json", "multi_node_tp": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/multi_node_tp.json", "multi_node_tp_pp": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/multi_node_tp_pp.json", "multi_node_tep": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/multi_node_tep.json", "multi_node_dep": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/multi_node_dep.json", "pd_cluster": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/meta-models/Muse-Spark-Open-816B-A42B/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "b200": "/meta-models/Muse-Spark-Open-816B-A42B/hw/b200.json", "gb200": "/meta-models/Muse-Spark-Open-816B-A42B/hw/gb200.json", "b300": "/meta-models/Muse-Spark-Open-816B-A42B/hw/b300.json", "gb300": "/meta-models/Muse-Spark-Open-816B-A42B/hw/gb300.json" } }, "model": { "model_id": "meta-models/Muse-Spark-Open-816B-A42B", "min_vllm_version": "0.33.0", "nightly_required": true, "docker_image": { "nvidia": "vllm/vllm-openai:muse-spark-open", "amd": "vllm/vllm-openai-rocm:muse-spark-open" }, "default_frontend": "rust", "architecture": "moe", "parameter_count": "816B", "active_parameters": "42B", "context_length": 1048576, "base_args": [], "base_env": {}, "install": { "docker": { "command": "docker pull vllm/vllm-openai:muse-spark-open", "note": "Serve from the image of your GPU vendor — no pip wheel is available now." } } }, "features": { "reasoning": { "description": "Channel-scoped chain-of-thought. Strength is set per request with the top-level `reasoning_effort` field — low/medium/high/xhigh map to strength hints 8/32/256/512, defaulting to xhigh when the field is omitted. Legacy `disable_reasoning` and `enable_thinking` kwargs are ignored.", "args": [ "--reasoning-parser", "muse" ] }, "tool_calling": { "description": "Channel-scoped tool calls. The parser is scoped to the tool channel, so an invocation echoed back inside ordinary assistant text is not re-parsed as a call. The template always offers `self` as a valid recipient.", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "muse" ] }, "spec_decoding": { "description": "DSpark speculative decoding, drafting up to seven tokens per pass. On NVIDIA, adaptive verification keeps only the draft slots worth verifying, so it stays on at large batch sizes; on AMD it is a fixed seven-token profile, best for latency-bound serving.", "default_mode": "dspark", "modes": { "dspark": { "label": "DSpark", "description": "Seven-token DSpark profile used for BF16, MXFP4, and NVFP4.", "variants": [ "bf16", "mxfp4", "default" ], "args": [ "--speculative-config", "{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":true}" ], "hardware_overrides": { "amd": { "args": [ "--speculative-config", "{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":false,\"attention_backend\":\"ROCM_AITER_FA\"}" ], "env": { "AITER_UNIFIED_ATTN_SHORT_QUERY_3D": "1", "VLLM_KV_CACHE_LAYOUT": "LBHNC" } } } } } } }, "opt_in_features": [ "spec_decoding" ], "variants": { "bf16": { "model_id": "meta-models/Muse-Spark-Open-816B-A42B", "precision": "bf16", "vram_minimum_gb": 1959, "description": "BF16 reference weights for NVIDIA and AMD; the ROCm path is validated on gfx950 (MI350X/MI355X).", "hardware_overrides": { "amd": { "extra_args": [ "--moe-backend", "triton", "--attention-backend", "ROCM_AITER_FA", "--max-model-len", "32768", "--max-num-seqs", "32", "--max-num-batched-tokens", "4096", "--gpu-memory-utilization", "0.95" ] } } }, "mxfp4": { "model_id": "amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4", "precision": "mxfp4", "vram_minimum_gb": 611, "supported_hardware": [ "mi350x", "mi355x" ], "description": "AMD Quark OCP MXFP4 W4A4 weights for gfx950 on the AITER MXFP4 MoE kernels.", "hardware_overrides": { "amd": { "extra_args": [ "--attention-backend", "ROCM_AITER_FA", "--max-model-len", "32768", "--max-num-seqs", "128", "--max-num-batched-tokens", "8192", "--gpu-memory-utilization", "0.90" ] } }, "json": "/amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4.json" }, "default": { "model_id": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4", "precision": "nvfp4", "vram_minimum_gb": 640, "description": "NVFP4 weights for Blackwell. The primary NVIDIA serving path.", "default_modes": { "spec_decoding": "dspark" }, "json": "/meta-models/Muse-Spark-Open-816B-A42B-NVFP4.json" } }, "strategy_hardware": { "single_node_dep": { "amd": "unsupported" }, "multi_node_dep": { "amd": "unsupported" } }, "hardware_overrides": { "amd": { "extra_args": [ "--dtype", "bfloat16", "--distributed-executor-backend", "mp", "--language-model-only", "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}", "--no-enable-prefix-caching", "--enable-chunked-prefill" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1", "VLLM_ROCM_MUSE_MOE_REDUCE_DTYPE": "bf16", "VLLM_USE_BREAKABLE_CUDAGRAPH": "1" } } }, "guide": "## Overview\n\nMuse Spark Open is a sparse-MoE reasoning model — 816B total parameters, 42B\nactive — with a 1M-token context window, aimed at long-horizon agentic coding.\nIt is the model Muse Glimmer 30B was distilled from.\n\n## Architecture\n\n62 layers, hidden size 8192, and **every layer is a mixture-of-experts layer** —\nthere are no dense layers. Each holds 256 experts and uses 8 per token. The\nexperts are 799B of the 816B; everything else is small.\n\n**The experts work narrow.** A learned matrix squeezes each token from 8192 to\n4096, all 256 experts run at that width, and a second matrix widens the result\nback. Both stay in full precision even when the experts are 4-bit. The router\nreads the token *before* the squeeze, and in fp32 — picking the wrong expert\ncosts more than running a smaller one. There is no shared expert.\n\n**Attention is 128 query heads sharing 16 key-value heads**, head dim 64, so the\nKV cache is eight times smaller than with one key-value head per query head.\n\n**Layers alternate between a narrow and a wide view**, on a cycle of four:\n\n```\nsliding_window: 2048\nlayer_types: [sliding_attention, full_attention, sliding_attention, sliding_attention, ...]\n```\n\nThree layers in each cycle look back 2048 tokens; the other (layers 1, 5, …,\n61) sees the whole context and drops rotary position (`layer_rope_theta: 0`),\nbecause rotary becomes unreliable over very long distances.\nAcross 62 layers that is **46 narrow layers and 16 wide ones**, and it is what\nmakes a 1M-token context affordable — only the 16 wide layers hold KV for the\nwhole sequence.\n\n**Normalization appears in more places than usual**, each doing a job: a norm\nbefore *and* after each block (the ones after carry no learnable weight), on\nqueries and keys before rotary position, on each attention head before its\nsigmoid gate, and on every expert's output before the weighted blend. The\nresidual stream is carried in fp32.\n\nLogits are scaled and then squashed through tanh (`final_logit_softcapping: 20.0`)\nbefore the softmax.\n\n## Prerequisites\n\n- **vLLM**: Use Docker image for now until the code is in vLLM main repo.\n- **Hardware**: the checkpoints hold ~1.64 TB of weights in BF16 and\n ~509 GB in Quark MXFP4, before KV cache. NVFP4 (~533 GB) is the primary\n NVIDIA Blackwell path and fits one 8-GPU node. On AMD, all three AMD\n variants are validated on gfx950 (MI350X/MI355X).\n\n## NVIDIA\n\nMeta ships BF16 and NVFP4 checkpoints. On Blackwell, serve\n`meta-models/Muse-Spark-Open-816B-A42B-NVFP4` on one 8-GPU node with\n**Tensor + Expert Parallel** (TP8 + `--enable-expert-parallel`), the default\nstrategy here. The command builder turns DSpark on by default for NVFP4; see\n[Speculative decoding](#speculative-decoding-dspark).\n\nNVIDIA supports every parallelism this recipe offers: TP, TP + PP, TP + expert\nparallel, data + expert parallel, and prefill/decode disaggregation. AMD\nsupports only TP and TP + expert parallel.\n\n## AMD ROCm variants\n\nQuark MXFP4 is the default AMD variant.\n\n| Variant | Checkpoint | AMD hardware | Experts / attention |\n| --- | --- | --- | --- |\n| MXFP4 | `amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4` | MI350X, MI355X | AITER MXFP4 MoE / AITER FA |\n| BF16 → MXFP4 | `meta-models/Muse-Spark-Open-816B-A42B` | MI350X, MI355X | experts of layers 1-60 rounded to MXFP4 at load, AITER MXFP4 MoE / AITER FA |\n| BF16 | `meta-models/Muse-Spark-Open-816B-A42B` | MI350X, MI355X | Triton BF16 MoE / AITER FA |\n\nIn both MXFP4 variants the experts of the first and last layers stay BF16 and\nrun on Triton BF16 MoE.\n\nOn AMD, the recipe's commands are text-only (`--language-model-only`), use the\nV2 model runner, disable prefix caching, enable chunked prefill, and request\nbreakable `FULL_AND_PIECEWISE` graphs. The ROCm MoE path supports the Tensor\nParallel and Tensor + Expert Parallel strategies; data-parallel strategies\nare not available on AMD.\n\n### Choosing TP or TP + expert parallel (MXFP4 variants)\n\nUse plain **TP** for latency-bound serving (a handful of concurrent requests)\nand the **Tensor + Expert Parallel** strategy (TP8 + `--enable-expert-parallel`,\n32 of 256 experts per GPU) for throughput at 32 or more concurrent requests,\nwhere it also lowers TTFT. Accuracy is the same under both. Keep **TP** for\nBF16: its Triton experts are slower with expert parallelism. With DSpark on,\nexpert parallelism pays off from a few concurrent requests; see the DSpark\nsection.\n\nThe AMD commands use `--max-model-len 32768`. For the full\n1M-token context, the MXFP4 variant is validated on TP with\n`--max-model-len 1048576` and `--max-num-seqs 8`, with either the default or\n`--kv-cache-dtype fp8` KV cache.\n\n## Launching the Server\n\nNVIDIA Blackwell, NVFP4, TP8 + expert parallel, with DSpark:\n\n```bash\nexport VLLM_USE_RUST_FRONTEND=1\n\nvllm serve meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \\\n --tensor-parallel-size 8 \\\n --enable-expert-parallel \\\n --max-model-len 1048576 \\\n --reasoning-parser muse \\\n --tool-call-parser muse \\\n --enable-auto-tool-choice \\\n --speculative-config '{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\",\"enable_adaptive_verification\":true}'\n```\n\n## Frontends\n\nCommands use the Rust frontend by default on NVIDIA and AMD ROCm.\n\nTo use the Python frontend, select Python in the command builder or replace\n`VLLM_USE_RUST_FRONTEND=1` with `VLLM_USE_RUST_FRONTEND=0` in the generated\ncommand.\n\n## Reasoning effort\n\nSet the strength per request with the top-level `reasoning_effort` field:\n\n| `reasoning_effort` | Strength hint |\n| --- | --- |\n| `low` | 8 |\n| `medium` | 32 |\n| `high` | 256 |\n| `xhigh` | 512 (default) |\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n -H 'Content-Type: application/json' \\\n -d '{\n \"model\": \"meta-models/Muse-Spark-Open-816B-A42B-NVFP4\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Refactor this module.\"}],\n \"reasoning_effort\": \"xhigh\"\n }'\n```\n\n### Multimodal\n\nMuse Spark is a native multimodal model that understands image, audio and video. Video is\nprocessed through a series of sampled video frames at load time. On AMD,\nthe recipe serves text only.\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n -H 'Content-Type: application/json' \\\n -d '{\n \"model\": \"meta-models/Muse-Spark-Open-816B-A42B-NVFP4\",\n \"messages\": [{\"role\": \"user\", \"content\": [\n {\"type\": \"video_url\", \"video_url\": {\"url\": \"https://example.com/demo.mp4\"}},\n {\"type\": \"text\", \"text\": \"What happens, and when?\"}\n ]}]\n }'\n```\n\nThe audio track of a video is not processed. To include it, extract the\naudio and send it as a separate audio part in the same request.\n\nVideo sampling rate can be changed when starting the server.\n\n```bash\n--media-io-kwargs '{\"video\": {\"fps\": 1}}'\n```\n\n## Speculative decoding (DSpark)\n\nInferact trained a DSpark speculator for this model. The draft is a\nsmall sliding-window model that reads the target's residual stream at selected\nlayers (`target_layer_ids`) and proposes a block per forward.\n\nOn NVIDIA, DSpark is on by default for NVFP4. It drafts up to seven tokens and\nuses **adaptive verification** (`enable_adaptive_verification: true`), which\nuses draft confidence to verify only the slots likely to be accepted. Because\nof that it keeps paying off at large batch sizes, not just for single-request\nlatency. Turn off **Spec Decoding** in the builder to serve without the\ndraft.\n\nOn AMD, selecting **DSpark** emits the fixed seven-token profile with AITER FA\nattention on every variant. Its causal multi-token decode is handled by\nAITER unified attention, with the gfx950 short-query 3D schedule enabled.\nDSpark gains the most when only a few requests are in flight, and still\nstays ahead of plain TP at 32 concurrent requests. It also runs under the\n**Tensor + Expert Parallel** strategy on the MXFP4 variants (TP8 +\n`--enable-expert-parallel`), which is the faster combination from a few\nconcurrent requests up to about 64 at the same accuracy; plain TP + DSpark\nremains the lowest-latency choice for a single request. With expert\nparallelism, DSpark is validated up to `--max-num-seqs 128`; with plain TP,\nkeep `--max-num-seqs 32`. For throughput beyond about 64 concurrent\nrequests, use expert parallelism without DSpark.\n\n### Adaptive DSpark (manual opt-in, Triton attention)\n\nAdaptive verification captures variable-length decode CUDA graphs, which\nthe AITER FA attention backend does not support; with the AMD MXFP4\ncommand above it fails at startup. To use it on MI350X/MI355X, switch the\ntarget to Triton attention and use the adaptive config:\n\n```bash\nexport VLLM_ADAPTIVE_VERIFICATION_PROFILE_CONTEXT_LEN=8192\n\n--attention-backend TRITON_ATTN \\\n--speculative-config '{\"method\":\"dspark\",\"model\":\"Inferact/Muse-Spark-Open-DSpark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"standard\",\"enable_adaptive_verification\":true,\"attention_backend\":\"TRITON_ATTN\"}'\n```\n\nSeven tokens remain the maximum draft depth. At startup, adaptive verification\nprofiles the DSpark and target costs, then uses draft confidence to keep only\nthe slots whose expected accepted-token benefit justifies their verification\ncost. There is no hand-tuned acceptance threshold. Keep the profile context\nrepresentative of production traffic.\n\n\n## References\n\n- [Muse Glimmer 30B recipe](/meta-models/Muse-Glimmer-30B) — the distilled 30B sibling\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/meta-models/Muse-Spark-Open-816B-A42B.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/meta-models/Muse-Spark-Open-816B-A42B.yaml", "overlays": [] } }