{
  "hf_id": "Qwen/Qwen3-32B",
  "meta": {
    "title": "Qwen3-32B",
    "slug": "qwen3-32b",
    "provider": "Qwen",
    "description": "Qwen3 32B dense model with hybrid thinking/non-thinking modes — verified on TPU v6e (Trillium).",
    "date_added": "2025-10-15",
    "date_updated": "2026-09-10",
    "tasks": [
      "text"
    ],
    "performance_headline": "Verified on TPU v6e (Trillium) and v7 (Ironwood) with BF16",
    "related_recipes": [
      "Qwen/Qwen3-4B"
    ],
    "hardware": {
      "trillium": "verified",
      "ironwood": "verified",
      "arc_pro_b60": "verified",
      "arc_pro_b70": "verified"
    }
  },
  "recommended_command": {
    "hardware": "h200",
    "strategy": "single_node_tp",
    "variant": "default",
    "node_count": 1,
    "deploy_type": "single_node",
    "env": {},
    "docker_image": "vllm/vllm-openai:latest",
    "command": "vllm serve Qwen/Qwen3-32B \\\n  --tensor-parallel-size 1 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser hermes \\\n  --reasoning-parser qwen3",
    "argv": [
      "vllm",
      "serve",
      "Qwen/Qwen3-32B",
      "--tensor-parallel-size",
      "1",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "hermes",
      "--reasoning-parser",
      "qwen3"
    ],
    "docker_command": "docker run --gpus all \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-openai:latest Qwen/Qwen3-32B \\\n  --tensor-parallel-size 1 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser hermes \\\n  --reasoning-parser qwen3",
    "docker_argv": [
      "docker",
      "run",
      "--gpus",
      "all",
      "--privileged",
      "--ipc=host",
      "-p",
      "8000:8000",
      "-v",
      "~/.cache/huggingface:/root/.cache/huggingface",
      "vllm/vllm-openai:latest",
      "Qwen/Qwen3-32B",
      "--tensor-parallel-size",
      "1",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "hermes",
      "--reasoning-parser",
      "qwen3"
    ],
    "strategy_spec": {
      "name": "single_node_tp",
      "deploy_type": "single_node",
      "display_name": "Tensor Parallel",
      "short_label": "TP",
      "orientation": "latency",
      "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n",
      "hardware_match": {
        "min_gpus": 1,
        "max_gpus": 8,
        "multi_node": false
      },
      "vllm_args": [],
      "parallel_flag": "--tensor-parallel-size"
    },
    "hardware_profile": {
      "brand": "NVIDIA",
      "generation": "hopper",
      "compute_arch": "sm90",
      "display_name": "H200",
      "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node",
      "gpu_count": 8,
      "vram_gb": 1128,
      "multi_node": false
    },
    "alternatives": {
      "multi_node_tp": "/Qwen/Qwen3-32B/strategies/multi_node_tp.json",
      "kv_store_centralized_mooncake": "/Qwen/Qwen3-32B/strategies/kv_store_centralized_mooncake.json",
      "kv_store_distributed_mooncake": "/Qwen/Qwen3-32B/strategies/kv_store_distributed_mooncake.json"
    },
    "by_hardware": {
      "h200": "/Qwen/Qwen3-32B/hw/h200.json",
      "h100": "/Qwen/Qwen3-32B/hw/h100.json",
      "b200": "/Qwen/Qwen3-32B/hw/b200.json",
      "gb200": "/Qwen/Qwen3-32B/hw/gb200.json",
      "b300": "/Qwen/Qwen3-32B/hw/b300.json",
      "gb300": "/Qwen/Qwen3-32B/hw/gb300.json",
      "mi300x": "/Qwen/Qwen3-32B/hw/mi300x.json",
      "mi325x": "/Qwen/Qwen3-32B/hw/mi325x.json",
      "mi350x": "/Qwen/Qwen3-32B/hw/mi350x.json",
      "mi355x": "/Qwen/Qwen3-32B/hw/mi355x.json",
      "trillium": "/Qwen/Qwen3-32B/hw/trillium.json",
      "ironwood": "/Qwen/Qwen3-32B/hw/ironwood.json",
      "arc_pro_b60": "/Qwen/Qwen3-32B/hw/arc_pro_b60.json",
      "arc_pro_b70": "/Qwen/Qwen3-32B/hw/arc_pro_b70.json"
    }
  },
  "model": {
    "model_id": "Qwen/Qwen3-32B",
    "min_vllm_version": "0.8.5",
    "architecture": "dense",
    "parameter_count": "32B",
    "active_parameters": "32B",
    "context_length": 40960,
    "base_args": [],
    "base_env": {},
    "install": {
      "pip": {
        "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto"
      },
      "docker": {
        "command": "docker pull vllm/vllm-openai:latest"
      }
    }
  },
  "features": {
    "tool_calling": {
      "description": "Enable automatic tool choice with the Hermes parser",
      "args": [
        "--enable-auto-tool-choice",
        "--tool-call-parser",
        "hermes"
      ]
    },
    "reasoning": {
      "description": "Enable Qwen3 thinking mode (chain-of-thought) with the Qwen3 reasoning parser",
      "args": [
        "--reasoning-parser",
        "qwen3"
      ]
    },
    "long_context": {
      "description": "YaRN RoPE scaling to the model card's validated 131,072-token context. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts — it can cost quality below 32K.",
      "args": [
        "--hf-overrides",
        "{\"max_position_embeddings\":131072,\"rope_parameters\":{\"rope_type\":\"yarn\",\"factor\":4.0,\"original_max_position_embeddings\":32768}}",
        "--max-model-len",
        "131072"
      ]
    },
    "spec_decoding": {
      "description": "Eagle3 speculative decoding with externally trained draft model; the served checkpoint is unchanged.",
      "args": [
        "--speculative-config",
        "{\"model\": \"RedHatAI/Qwen3-32B-speculator.eagle3\", \"num_speculative_tokens\": 3, \"method\": \"eagle3\"}"
      ]
    }
  },
  "opt_in_features": [
    "spec_decoding",
    "long_context"
  ],
  "variants": {
    "default": {
      "precision": "bf16",
      "vram_minimum_gb": 77,
      "description": "Full precision BF16 — TPU v6e 2x2 (4 chips, TP=4) or a single 80GB+ GPU"
    },
    "fp8": {
      "model_id": "Qwen/Qwen3-32B-FP8",
      "precision": "fp8",
      "vram_minimum_gb": 39,
      "description": "Qwen official FP8 checkpoint — single 40 GB GPU",
      "json": "/Qwen/Qwen3-32B-FP8.json"
    },
    "awq": {
      "model_id": "Qwen/Qwen3-32B-AWQ",
      "precision": "int4",
      "vram_minimum_gb": 20,
      "description": "AWQ 4-bit quantized weights — fits on a single 24 GB GPU",
      "extra_args": [
        "--quantization",
        "awq"
      ],
      "json": "/Qwen/Qwen3-32B-AWQ.json"
    },
    "redhat_fp8": {
      "model_id": "RedHatAI/Qwen3-32B-FP8-dynamic",
      "precision": "fp8",
      "label": "FP8 Dynamic (RedHat)",
      "vram_minimum_gb": 39,
      "description": "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization",
      "extra_args": [
        "--kv-cache-dtype",
        "fp8"
      ],
      "json": "/RedHatAI/Qwen3-32B-FP8-dynamic.json"
    },
    "redhat_fp8_block": {
      "model_id": "RedHatAI/Qwen3-32B-FP8-block",
      "precision": "fp8",
      "label": "FP8 Block (RedHat)",
      "vram_minimum_gb": 39,
      "description": "RedHatAI FP8 block quantization",
      "extra_args": [
        "--kv-cache-dtype",
        "fp8"
      ],
      "json": "/RedHatAI/Qwen3-32B-FP8-block.json"
    },
    "nvfp4": {
      "model_id": "RedHatAI/Qwen3-32B-NVFP4",
      "precision": "nvfp4",
      "vram_minimum_gb": 22,
      "description": "RedHatAI NVFP4 quantization",
      "json": "/RedHatAI/Qwen3-32B-NVFP4.json"
    },
    "redhat_nvfp4": {
      "model_id": "RedHatAI/Qwen3-32B-NVFP4A16",
      "precision": "nvfp4",
      "label": "NVFP4 A16 (RedHat)",
      "vram_minimum_gb": 22,
      "description": "RedHatAI NVFP4 with BF16 activations",
      "json": "/RedHatAI/Qwen3-32B-NVFP4A16.json"
    }
  },
  "kv_offload_support": {
    "offloading_cpu": "verified",
    "offloading_fs": "verified"
  },
  "hardware_overrides": {
    "xpu": {
      "extra_args": [
        "--gpu-memory-utilization",
        "0.90"
      ]
    }
  },
  "guide": "## Overview\n\nQwen3-32B is the flagship dense model in the Qwen3 series, with hybrid thinking / non-thinking modes. BF16 is the training precision and gives the best accuracy.\n\nToggle thinking mode per request with `enable_thinking` in `chat_template_kwargs`, or set the **Reasoning** feature to add the Qwen3 reasoning parser so `<think>` content is split into a separate field.\n\n## Prerequisites\n\n### NVIDIA\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto\n```\n\n### TPU (Trillium / v6e)\nUse the official vLLM TPU image. The 32B model is served on a `v6e-4` slice (`--topology 2x2`, 4 chips, TP=4).\n```bash\nexport HF_HOME=/dev/shm\nexport HF_TOKEN=<your HF token>\n```\n\n### Intel XPU (B60 / B70)\nUse the official vLLM XPU image.\n```bash\ndocker pull vllm/vllm-openai-xpu:latest\n```\n\n## Deployment Configurations\n\n### TPU v6e (Trillium, 2x2 slice, TP=4)\nVerified end-to-end on a 4-chip v6e slice.\n```bash\nexport HF_HOME=/dev/shm\nexport HF_TOKEN=<your HF token>\nexport TP=4\nexport MAX_MODEL_LEN=4096\nvllm serve Qwen/Qwen3-32B \\\n  --seed 42 \\\n  --gpu-memory-utilization 0.98 \\\n  --max-num-batched-tokens 2048 \\\n  --max-num-seqs 256 \\\n  --tensor-parallel-size $TP \\\n  --max-model-len $MAX_MODEL_LEN\n```\n\n### TPU v7 (Ironwood, single host, TP=2)\nVerified end-to-end on a single Ironwood host (`tpu7x-standard-1t`, topology `1x1x1`). Served via the OpenAI API server entrypoint.\n```bash\nexport HF_HOME=/data\nexport HUGGING_FACE_HUB_TOKEN=<your HF token>\nexport LAYOUT_Q_PROJ_AS_NDH=1\nexport USE_BATCHED_RPA_KERNEL=1\nvllm serve Qwen/Qwen3-32B \\\n  --host 0.0.0.0 \\\n  --port 8000 \\\n  --seed 42 \\\n  --tensor-parallel-size 2 \\\n  --data-parallel-size 1 \\\n  --max-model-len 9216 \\\n  --max-num-batched-tokens 4096 \\\n  --max-num-seqs 135 \\\n  --gpu-memory-utilization 0.98 \\\n  --block-size 256 \\\n  --kv-cache-dtype fp8 \\\n  --no-enable-prefix-caching \\\n  --async-scheduling\n```\n> Ironwood support ships in the nightly TPU image (e.g. `vllm/vllm-tpu:nightly-20260330-2f76400-8c0b626`).\n\n### Single node (BF16, TP=2)\n```bash\nvllm serve Qwen/Qwen3-32B \\\n  --host 0.0.0.0 \\\n  --port 8000 \\\n  --tensor-parallel-size 2\n```\n\n### Intel XPU (B60, B70)\n\nValidated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the\nofficial vLLM XPU image `vllm/vllm-openai-xpu:latest`.\n\n```bash\ndocker run --device /dev/dri \\\n  -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-openai-xpu:latest \\\n  Qwen/Qwen3-32B \\\n  --gpu-memory-utilization 0.90 \\\n  --tensor-parallel-size 4 \\\n  --max-model-len 8192\n```\n\n## Configuration Tips\n\n- `--max-model-len 4096` keeps memory in check on a 4-chip v6e slice; native context is 40K — raise it if you have headroom.\n- vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.\n- `--max-num-batched-tokens` and `--max-num-seqs` tune batch throughput vs latency.\n- For thinking mode Qwen recommends `temperature=0.6, top_p=0.95, top_k=20`; for non-thinking, `temperature=0.7, top_p=0.8`.\n\n## Benchmarking\n\nLaunch the server with `--no-enable-prefix-caching` for consistent measurements.\n\n```bash\nvllm bench serve \\\n  --host 0.0.0.0 \\\n  --port 8000 \\\n  --model Qwen/Qwen3-32B \\\n  --dataset-name random \\\n  --random-input-len 1024 \\\n  --random-output-len 1024 \\\n  --num-prompts 128\n```\n\n## References\n\n- [Model card](https://huggingface.co/Qwen/Qwen3-32B)\n- [Qwen3 on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3)\n- [Qwen3-32B on Ironwood (v7)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-32B)\n",
  "source": {
    "source": "vllm",
    "repo": "vllm-project/vllm",
    "ref": "main",
    "sha": "389ffebe555705c85337025d5df108d0384d365e",
    "path": "recipes/models/Qwen/Qwen3-32B.yaml",
    "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/Qwen/Qwen3-32B.yaml",
    "overlays": []
  }
}