# Qwen/Qwen3-32B — vLLM serving recipe

> Qwen3 32B dense model with hybrid thinking/non-thinking modes — verified on TPU v6e (Trillium).

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/Qwen/Qwen3-32B
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/Qwen/Qwen3-32B.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`, `trillium`, `ironwood`, `arc_pro_b60`, `arc_pro_b70`
- `variant`: `default`, `fp8`, `awq`, `redhat_fp8`, `redhat_fp8_block`, `nvfp4`, `redhat_nvfp4`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `long_context`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.8.5.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve Qwen/Qwen3-32B \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --reasoning-parser qwen3
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest Qwen/Qwen3-32B \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --reasoning-parser qwen3
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Qwen3-32B is the flagship dense model in the Qwen3 series, with hybrid thinking / non-thinking modes. BF16 is the training precision and gives the best accuracy.

Toggle thinking mode per request with `enable_thinking` in `chat_template_kwargs`, or set the **Reasoning** feature to add the Qwen3 reasoning parser so `<think>` content is split into a separate field.

### Prerequisites

#### NVIDIA
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### TPU (Trillium / v6e)
Use the official vLLM TPU image. The 32B model is served on a `v6e-4` slice (`--topology 2x2`, 4 chips, TP=4).
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
```

#### Intel XPU (B60 / B70)
Use the official vLLM XPU image.
```bash
docker pull vllm/vllm-openai-xpu:latest
```

### Deployment Configurations

#### TPU v6e (Trillium, 2x2 slice, TP=4)
Verified end-to-end on a 4-chip v6e slice.
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
export TP=4
export MAX_MODEL_LEN=4096
vllm serve Qwen/Qwen3-32B \
  --seed 42 \
  --gpu-memory-utilization 0.98 \
  --max-num-batched-tokens 2048 \
  --max-num-seqs 256 \
  --tensor-parallel-size $TP \
  --max-model-len $MAX_MODEL_LEN
```

#### TPU v7 (Ironwood, single host, TP=2)
Verified end-to-end on a single Ironwood host (`tpu7x-standard-1t`, topology `1x1x1`). Served via the OpenAI API server entrypoint.
```bash
export HF_HOME=/data
export HUGGING_FACE_HUB_TOKEN=<your HF token>
export LAYOUT_Q_PROJ_AS_NDH=1
export USE_BATCHED_RPA_KERNEL=1
vllm serve Qwen/Qwen3-32B \
  --host 0.0.0.0 \
  --port 8000 \
  --seed 42 \
  --tensor-parallel-size 2 \
  --data-parallel-size 1 \
  --max-model-len 9216 \
  --max-num-batched-tokens 4096 \
  --max-num-seqs 135 \
  --gpu-memory-utilization 0.98 \
  --block-size 256 \
  --kv-cache-dtype fp8 \
  --no-enable-prefix-caching \
  --async-scheduling
```
> Ironwood support ships in the nightly TPU image (e.g. `vllm/vllm-tpu:nightly-20260330-2f76400-8c0b626`).

#### Single node (BF16, TP=2)
```bash
vllm serve Qwen/Qwen3-32B \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2
```

#### Intel XPU (B60, B70)

Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.

```bash
docker run --device /dev/dri \
  -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-xpu:latest \
  Qwen/Qwen3-32B \
  --gpu-memory-utilization 0.90 \
  --tensor-parallel-size 4 \
  --max-model-len 8192
```

### Configuration Tips

- `--max-model-len 4096` keeps memory in check on a 4-chip v6e slice; native context is 40K — raise it if you have headroom.
- vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.
- `--max-num-batched-tokens` and `--max-num-seqs` tune batch throughput vs latency.
- For thinking mode Qwen recommends `temperature=0.6, top_p=0.95, top_k=20`; for non-thinking, `temperature=0.7, top_p=0.8`.

### Benchmarking

Launch the server with `--no-enable-prefix-caching` for consistent measurements.

```bash
vllm bench serve \
  --host 0.0.0.0 \
  --port 8000 \
  --model Qwen/Qwen3-32B \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 128
```

### References

- [Model card](https://huggingface.co/Qwen/Qwen3-32B)
- [Qwen3 on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3)
- [Qwen3-32B on Ironwood (v7)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-32B)