# zai-org/GLM-4.5V — vLLM serving recipe

> GLM-4.5 vision-language MoE model (~107B parameters, BF16) with image-text-to-text capability, 64K context, expert parallelism, and native FP8

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`, `encoder_parallel`

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/zai-org/GLM-4.5V
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/zai-org/GLM-4.5V.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_tep`, `multi_node_dep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `text_only`, `encoder_parallel` (default: `tool_calling`, `reasoning`, `encoder_parallel`)

## Install

Requires vLLM >= 0.12.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

Extra dependencies:

- GLM-V requires Transformers >= 5.0.0 for tokenizer and multimodal config compatibility.

```bash
uv pip install "transformers>=5.0.0"
```

## Serve

```bash
vllm serve zai-org/GLM-4.5V \
  --trust-remote-code \
  --enable-expert-parallel \
  --allowed-local-media-path / \
  --mm-processor-cache-type shm \
  --tensor-parallel-size 8 \
  --tool-call-parser glm45 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45 \
  --mm-encoder-tp-mode data
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest zai-org/GLM-4.5V \
  --trust-remote-code \
  --enable-expert-parallel \
  --allowed-local-media-path / \
  --mm-processor-cache-type shm \
  --tensor-parallel-size 8 \
  --tool-call-parser glm45 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45 \
  --mm-encoder-tp-mode data
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

GLM-4.5V is the vision-language variant of GLM-4.5. It is an MoE model with ~107B
total parameters that accepts image and text inputs. FP8 models have minimal accuracy
loss versus BF16 and are recommended for cost-efficient serving. GLM-4.5V supports a
64K context length (use GLM-4.6V for 128K).

### Prerequisites

- **vLLM version:** >= 0.12.0
- **Hardware:** 4x H100/H200 (BF16 or FP8), or 4x MI300X/MI325X/MI355X with ROCm
- **Python:** 3.10+ for NVIDIA; Python 3.12 with ROCm 7.0 and glibc >= 2.35 for AMD ROCm wheels

#### Install vLLM (NVIDIA)

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### Install vLLM (AMD ROCm)

```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm
uv pip install "transformers>=5.0.0"
```

### Launching the Server

#### Tensor Parallel + Expert Parallel (FP8 on 4 GPUs)

```bash
vllm serve zai-org/GLM-4.5V-FP8 \
     --tensor-parallel-size 4 \
     --tool-call-parser glm45 \
     --reasoning-parser glm45 \
     --enable-auto-tool-choice \
     --enable-expert-parallel \
     --allowed-local-media-path / \
     --mm-encoder-tp-mode data \
     --mm-processor-cache-type shm
```

#### AMD ROCm (4xMI300x/MI325x/MI355x))

```bash
export VLLM_ROCM_USE_AITER=1

vllm serve zai-org/GLM-4.5V-FP8 \
     --tensor-parallel-size 4 \
     --tool-call-parser glm45 \
     --reasoning-parser glm45 \
     --enable-auto-tool-choice \
     --enable-expert-parallel \
     --allowed-local-media-path / \
     --mm-encoder-tp-mode data \
     --mm-processor-cache-type shm
```

#### Tuning Tips

- `--max-model-len=65536` is near the model's max context (64K).
- `--max-num-batched-tokens=32768` for prompt-heavy workloads.
- `--gpu-memory-utilization=0.95` maximizes KV cache.
- Use `uv pip install "transformers>=5.0.0"` before serving GLM-V checkpoints.
- `--mm-encoder-tp-mode data` runs the vision encoder data-parallel — preferable to TP
  since the encoder is small and TP adds communication overhead.
- `--mm-processor-cache-type shm` enables shared-memory caching for repeated image inputs.

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="zai-org/GLM-4.5V-FP8",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "https://example.com/image.png"}},
            {"type": "text", "text": "Describe the image."}
        ]
    }],
    max_tokens=512,
)
print(resp.choices[0].message.content)
```

### Benchmarking

```bash
vllm bench serve \
  --backend openai-chat \
  --endpoint /v1/chat/completions \
  --model zai-org/GLM-4.5V-FP8 \
  --dataset-name hf \
  --dataset-path lmarena-ai/VisionArena-Chat \
  --num-prompts 1000 \
  --request-rate 20
```

### Troubleshooting

- **Vision encoder overhead:** Use `--mm-encoder-tp-mode data` unless TP-sharded encoder is known-good for your config.
- **Context length errors:** GLM-4.5V max is 64K; use GLM-4.6V if you need 128K.

### References

- [Model card](https://huggingface.co/zai-org/GLM-4.5V)
- [FP8 checkpoint](https://huggingface.co/zai-org/GLM-4.5V-FP8)
- [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)