# zai-org/GLM-OCR — vLLM serving recipe

> GLM-OCR image-to-text model with built-in MTP speculative decoding for high-throughput OCR serving

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `spec_decoding`

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/zai-org/GLM-OCR
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/zai-org/GLM-OCR.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `spec_decoding`, `text_only`, `encoder_parallel` (default: `spec_decoding`)

## Install

Requires vLLM >= 0.16.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

Extra dependencies:

- GLM-OCR requires the nightly vllm wheel

```bash
uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
```

- transformers from source for GLM-OCR tokenizer support

```bash
uv pip install git+https://github.com/huggingface/transformers.git
```

## Serve

```bash
vllm serve zai-org/GLM-OCR \
  --tensor-parallel-size 1 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 1
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest zai-org/GLM-OCR \
  --tensor-parallel-size 1 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 1
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

GLM-OCR is a vision-language model for end-to-end OCR. It includes built-in
Multi-Token Prediction (MTP) layers enabling speculative decoding for higher
throughput generation.

### Prerequisites

- **vLLM version:** nightly recommended (or latest stable with MTP support)
- **Transformers:** >= 5.0.0 (install from source for latest)

#### Install vLLM

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Or nightly:

```bash
uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
uv pip install git+https://github.com/huggingface/transformers.git
```

### Launching the Server

#### With MTP Speculative Decoding

```bash
vllm serve zai-org/GLM-OCR \
     --speculative-config.method mtp \
     --speculative-config.num_speculative_tokens 1
```

### Client Usage

#### OpenAI SDK

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)

messages = [{
    "role": "user",
    "content": [
        {"type": "image_url", "image_url": {"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"}},
        {"type": "text", "text": "Text Recognition:"}
    ]
}]

response = client.chat.completions.create(
    model="zai-org/GLM-OCR",
    messages=messages,
    max_tokens=2048,
    temperature=0.0,
)
print(response.choices[0].message.content)
```

#### cURL

```bash
curl -s http://localhost:8000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
          "model": "zai-org/GLM-OCR",
          "messages": [{
               "role": "user",
               "content": [
                    {"type": "image_url", "image_url": {"url": "https://example.com/receipt.png"}},
                    {"type": "text", "text": "Text Recognition:"}
               ]
          }],
          "max_tokens": 2048,
          "temperature": 0.0
     }'
```

### Troubleshooting

- **Greedy sampling recommended:** Use `temperature=0.0` for optimal OCR accuracy.
- **Transformers version:** Requires `transformers >= 5.0.0`.

### References

- [Model card](https://huggingface.co/zai-org/GLM-OCR)