# zai-org/GLM-5.3 — vLLM serving recipe

> GLM-5.3 — Frontier Coding with Emergent Cyber Capabilities

Configuration: hardware `b300` (8× B300) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (FP8) · features `tool_calling`, `reasoning` · frontend `rust`

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/zai-org/GLM-5.3
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/zai-org/GLM-5.3.json?hardware=b300

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `dgx_station_gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`, `ascend_950pr_8x`
- `variant`: `default`, `nvfp4`, `w4a8`, `bf16`
- `strategy`: `single_node_tp`, `single_node_tep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_tep`, `multi_node_dep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.29.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve zai-org/GLM-5.3 \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 8 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm47
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:latest zai-org/GLM-5.3 \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 8 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm47
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

**GLM-5.3** is the newest model in the GLM-5 series — a ~743B-parameter MoE (39B active) from **Z-AI**.
Architecture and serving flags are identical to GLM-5.2; the packaging change is that the
**default `zai-org/GLM-5.3` checkpoint is now native FP8** — BF16 weights live under the suffixed `zai-org/GLM-5.3-BF16` repo.

The FP8 checkpoint fits on a single 8xH200 / 8xH20 node and — with FP8 KV cache — reaches the full 1M-token context on 8xB200.
An NVFP4 checkpoint for Blackwell is published by Inferact (`Inferact/GLM-5.3-NVFP4`).

### Prerequisites

- **vLLM 0.28.0** or newer.
- **GPU:** 8xH200 or 8xH20 (141 GB each) for single-node FP8; 8xB200 (180 GB each) for the full 1M context.

### Installation

```bash
uv venv
source .venv/bin/activate
uv pip install "vllm==0.28.0" --torch-backend=auto
uv pip install "transformers>=5.15.0"
```

### Launching the server

#### FP8 on 8xH200 (standard)

```bash
vllm serve zai-org/GLM-5.3 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 5 \
  --tool-call-parser glm47 \
  --reasoning-parser glm47 \
  --enable-auto-tool-choice \
  --served-model-name glm-5.3
```

#### FP8 on AMD MI300X/MI325X/MI350X/MI355X

```bash
VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \
vllm serve zai-org/GLM-5.3 \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 5 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm47 \
  --gpu-memory-utilization 0.80 \
  --max-model-len 524288 \
  --max-num-seqs 32 \
  --linear-backend aiter \
  --moe-backend aiter
```

On 8x MI325X with `vllm/vllm-openai-rocm:v0.29.0`, this command serves the full
`--max-model-len 524288` at `--gpu-memory-utilization 0.80`, reporting a 1,477,168-token
KV cache and 2.82x maximum concurrency. MTP is healthy there: a mean acceptance length of
2.77 with 5 tokens drafted per step on a ~500-token greedy generation.

On 8x MI350X with the same image and settings, it reports a 2,013,072-token KV cache and
3.84x maximum concurrency, and MTP's mean acceptance length is 3.19 with 5 tokens drafted
per step on a 512-token greedy generation.

#### FP8 on 8xB200 (full 1M context)

GLM-5.3 has a native 1M-token window. Whether the full window fits is a KV-cache VRAM
question, so the lever is `--max-num-seqs` — it bounds how many sequences share the KV
budget at once. **Start at 32** and scale with your node's VRAM. FP8 KV cache
(`--kv-cache-dtype fp8_e4m3`, already in the base flags) roughly halves that budget.

```bash
vllm serve zai-org/GLM-5.3 \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 5 \
  --max-num-seqs 32 \
  --tool-call-parser glm47 \
  --reasoning-parser glm47 \
  --enable-auto-tool-choice \
  --served-model-name glm-5.3
```

- **`--max-num-seqs 32`** — the single knob for fitting 1M context; start at 32 and tune it
  to your VRAM (up on headroom, down on OOM).
- **BF16** weights are served from `zai-org/GLM-5.3-BF16` and need multi-node deployment.

#### NVFP4 on Blackwell (B200/B300)

The `Inferact/GLM-5.3-NVFP4` variant is Inferact's NVFP4 re-quantization : only the MoE expert linears drop to NVFP4 while shared experts,
attention, embeddings, and the early dense layers stay BF16. The ~465 GB checkpoint fits comfortably on Blackwell
Select the **NVFP4** variant above (Blackwell-only) or run:

```bash
vllm serve Inferact/GLM-5.3-NVFP4 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --reasoning-parser glm47 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --kv-cache-dtype fp8_e4m3 \
  --served-model-name glm-5.3-nvfp4
```

#### DSpark (opt-in)

DSpark is an external confidence-scheduled drafter. Serve `zai-org/GLM-5.3` and attach
[RedHatAI/GLM-5.3-speculator.dspark](https://huggingface.co/RedHatAI/GLM-5.3-speculator.dspark)
with eight speculative tokens:

```bash
vllm serve zai-org/GLM-5.3 \
  --tensor-parallel-size 8 \
  --speculative-config '{"method":"dspark","model":"RedHatAI/GLM-5.3-speculator.dspark","num_speculative_tokens":8}'
```

### Reasoning modes

Thinking is always on — the generation prompt opens a `<think>` block unconditionally.
GLM-5.3 offers **three reasoning effort levels** driven by the `reasoning_effort`
field; the default is **`max`**:

| Mode | How to request | Behavior |
|------|----------------|----------|
| **Think Max** (default) | omit `reasoning_effort`, or set `"max"` | Deepest reasoning — hard math, multi-step planning, agentic tasks. Highest token cost. |
| **Think High** | `"reasoning_effort": "high"` | Balanced depth and latency. |
| **Think Low** | `"reasoning_effort": "low"` | Lightest reasoning — simple Q&A, lowest latency and token cost. |

The chat template resolves effort to **`max` unless `reasoning_effort` is explicitly
`"low"` or `"high"`** (any other value falls back to `max`), then injects
`Reasoning Effort: Low|High|Max` into the system prompt. Pass it through
`chat_template_kwargs` or the top-level OpenAI `reasoning_effort` field.

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
msgs = [{"role": "user", "content": "Summarize GLM-5.3 in one sentence."}]

# Think Max (default) — just omit reasoning_effort
client.chat.completions.create(model="glm-5.3", messages=msgs, max_tokens=4096)

# Think High / Think Low — explicitly request the effort level
client.chat.completions.create(
    model="glm-5.3",
    messages=msgs,
    max_tokens=4096,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "low"}},
)
```

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-5.3",
    "messages": [{"role": "user", "content": "Summarize GLM-5.3 in one sentence."}],
    "temperature": 1,
    "max_tokens": 4096,
    "chat_template_kwargs": {"reasoning_effort": "high"}
  }'
```

### Client usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
response = client.chat.completions.create(
    model="glm-5.3",
    messages=[{"role": "user", "content": "Summarize GLM-5.3 in one sentence."}],
    temperature=1.0,
    max_tokens=256,
)
print(response.choices[0].message.content)
```

### Benchmarking

Add `--no-enable-prefix-caching` to the server command for a clean measurement.

```bash
vllm bench serve \
  --model zai-org/GLM-5.3 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1024 \
  --request-rate 10 \
  --num-prompts 32 \
  --ignore-eos
```

> **Note:** pure throughput benchmarks tend to under-report real speed, because MTP's
> acceptance rate is usually low in synthetic runs.

### Troubleshooting

- **FP8 performance:** DeepGEMM is required — install via `install_deepgemm.sh`.

### References

- [Model card](https://huggingface.co/zai-org/GLM-5.3)
- [BF16 checkpoint](https://huggingface.co/zai-org/GLM-5.3-BF16)
- [NVFP4 checkpoint (Inferact)](https://huggingface.co/Inferact/GLM-5.3-NVFP4)
- [DSpark draft](https://huggingface.co/RedHatAI/GLM-5.3-speculator.dspark)