# Qwen/Qwen3-Coder-480B-A35B-Instruct — vLLM serving recipe

> Large coder MoE with 480B total / 35B active parameters, strong tool-use and code generation capabilities.

Configuration: hardware `h200` (8× H200) · strategy `multi_node_tp` (Multi-Node Tensor Parallel) · 2 nodes · variant `default` (BF16) · features `tool_calling`

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/Qwen/Qwen3-Coder-480B-A35B-Instruct
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/Qwen/Qwen3-Coder-480B-A35B-Instruct.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`, `ironwood`
- `variant`: `default`, `fp8`, `nvfp4`, `redhat_fp8`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_dep`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling` (default: `tool_calling`)

## Install

Requires vLLM >= 0.10.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

Extra dependencies:

- Optional: opt into DeepGEMM FP8 MoE kernels for extra throughput (optional)

```bash
export VLLM_USE_DEEP_GEMM=1
```

## Serve

Run one command per node: the head (node rank 0) plus 1 worker. Each node's command differs — don't replay one on every node.

### Head (node 0)

```bash
export GLOO_SOCKET_IFNAME=$IFACE_NAME
export NCCL_SOCKET_IFNAME=$IFACE_NAME
export VLLM_USE_DEEP_GEMM=1
vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_IP \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
```

### Node 1

```bash
export GLOO_SOCKET_IFNAME=$IFACE_NAME
export NCCL_SOCKET_IFNAME=$IFACE_NAME
export VLLM_USE_DEEP_GEMM=1
vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 1 \
  --master-addr $HEAD_IP \
  --headless \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
```

Docker equivalent:

```bash
# Head
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e GLOO_SOCKET_IFNAME=$IFACE_NAME \
  -e NCCL_SOCKET_IFNAME=$IFACE_NAME \
  -e VLLM_USE_DEEP_GEMM=1 \
  vllm/vllm-openai:latest Qwen/Qwen3-Coder-480B-A35B-Instruct \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_IP \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Qwen3-Coder](https://github.com/QwenLM/Qwen3-Coder) is an advanced large language model created by the Qwen team. `Qwen3-Coder-480B-A35B-Instruct` is the flagship coder MoE with 480B total / 35B active parameters. vLLM supports it including tool calling; the guide below covers BF16 and FP8 serving on NVIDIA and AMD GPUs.

TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with a recipe for [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-Coder-480B-A35B). The Ironwood docker command rendered by the hardware picker uses the `vllm/vllm-tpu` image; pin to the tag specified by the upstream recipe.

### Prerequisites

#### CUDA
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### ROCm (MI300X, MI325X, MI355X)
```bash
uv venv
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```
The ROCm wheel requires Python 3.12, ROCm 7.0, and glibc >= 2.35.

### Deployment Configurations

#### 8xH200 / 8xH20 BF16
```bash
vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct \
  --max-model-len 32000 \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
```

#### 8xH200 / 8xH20 FP8 (DP=8, recommended)
```bash
VLLM_USE_DEEP_GEMM=1 vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
  --max-model-len 131072 \
  --enable-expert-parallel \
  --data-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
```

#### 8xMI300X/MI325X/MI355X BF16
```bash
VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct \
  --max-model-len 32000 \
  --attention-backend rocm_aiter_fa \
  --moe-backend aiter \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
```

#### 8xMI300X/MI325X/MI355X FP8
```bash
VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
  --trust-remote-code \
  --max-model-len 131072 \
  --attention-backend rocm_aiter_fa \
  --moe-backend aiter \
  --enable-expert-parallel \
  --data-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
```

### Client Usage

```bash
vllm bench serve \
  --backend vllm \
  --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
  --endpoint /v1/completions \
  --dataset-name random \
  --random-input 2048 \
  --random-output 1024 \
  --max-concurrency 10 \
  --num-prompt 100
```

### Troubleshooting

- **Context-length OOM**: A single H20 node cannot serve the native 262144 context. Reduce `--max-model-len` or raise `--gpu-memory-utilization`.
- **TP=8 failure on FP8**: Expect `ValueError: The output_size of gate's and up's weight = 320 is not divisible by weight quantization block_n = 128.` on FP8 with TP=8. Switch to `--data-parallel-size 8` instead.
- **DeepGEMM**: set `VLLM_USE_DEEP_GEMM=1` for faster FP8 matmul. Follow the [setup instructions](https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/deepgemm/README.md#setup) to install it.
- **Tool calling**: add `--tool-call-parser qwen3_coder` as shown above.

### Evaluation

| Dataset    | Test Type          | Pass@1 Score |
|------------|--------------------|--------------|
| HumanEval  | Base tests         | 0.939        |
| HumanEval+ | Base + extra tests | 0.902        |
| MBPP       | Base tests         | 0.918        |
| MBPP+      | Base + extra tests | 0.794        |

### References

- [Model card](https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct)
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8)
- [Qwen3-Coder GitHub](https://github.com/QwenLM/Qwen3-Coder)
- [EvalPlus](https://github.com/evalplus/evalplus)
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-Coder-480B-A35B)