# Qwen/Qwen3-235B-A22B-Instruct-2507 — vLLM serving recipe

> Flagship Qwen3 MoE instruct model with 235B total and 22B active parameters, tuned for high-quality text generation.

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/Qwen/Qwen3-235B-A22B-Instruct-2507
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/Qwen/Qwen3-235B-A22B-Instruct-2507.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`, `nvfp4`, `redhat_nvfp4`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_dep`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `spec_decoding` (default: `tool_calling`)

## Install

Requires vLLM >= 0.10.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Qwen3](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) is the flagship instruct MoE model in the Qwen3 series, with 235B total parameters and 22B active parameters. This guide covers deploying the model efficiently with vLLM on NVIDIA and AMD GPUs.

### Prerequisites

#### NVIDIA CUDA (pip)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend=auto
```

#### AMD ROCm (pip)
> Note: The vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35. Use the Docker flow if your environment is incompatible.
```bash
uv venv
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700
```

### Deployment Configurations

#### BF16 on MI300X/MI325X/MI355X (4 GPUs)

```bash
HIP_VISIBLE_DEVICES="4,5,6,7" \
VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_MHA=0 \
SAFETENSORS_FAST_GPU=1 \
vllm serve Qwen/Qwen3-235B-A22B \
  --trust-remote-code \
  -tp 4 \
  --swap-space 32 \
  --distributed-executor-backend mp \
  --max-num-batched-tokens 32768 \
  --max-model-len 32768 \
  --no-enable-prefix-caching \
  --gpu-memory-utilization 0.8
```

#### FP8 on MI300X/MI325X/MI355X (4 GPUs)

```bash
HIP_VISIBLE_DEVICES="4,5,6,7" \
VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_MHA=0 \
SAFETENSORS_FAST_GPU=1 \
vllm serve Qwen/Qwen3-235B-A22B-FP8 \
  --trust-remote-code \
  -tp 4 \
  --swap-space 16 \
  --distributed-executor-backend mp \
  --max-num-batched-tokens 32768 \
  --max-model-len 32768 \
  --no-enable-prefix-caching \
  --gpu-memory-utilization 0.8
```

#### TPU Deployment

- [Qwen3-32B on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3)
- [Qwen3-4B on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3)

### Client Usage

```bash
vllm bench serve \
  --model "Qwen/Qwen3-235B-A22B-FP8" \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 1024 \
  --request-rate 10000 \
  --num-prompts 16 \
  --ignore-eos \
  --trust-remote-code
```

### Troubleshooting

- Use `--max-num-batched-tokens` and `--max-model-len` to fit memory constraints on smaller nodes.
- If OOM at startup, lower `--gpu-memory-utilization` (e.g. 0.8) and disable prefix caching.
- AMD builds require `VLLM_ROCM_USE_AITER=1` for best performance.

### References

- [Model card](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507)
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3-235B-A22B-FP8)
- [vLLM documentation](https://docs.vllm.ai/)