# Qwen/Qwen2.5-VL-7B-Instruct — vLLM serving recipe

> Qwen2.5-VL dense vision-language model (7B) for image and video understanding — fits on a single TPU v6e chip, one GPU, or Intel Xeon 6 CPU.

Configuration: hardware `trillium` (8× TPU v6e (Trillium)) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `encoder_parallel`

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/Qwen/Qwen2.5-VL-7B-Instruct
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/Qwen/Qwen2.5-VL-7B-Instruct.json?hardware=trillium

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `trillium`, `xeon6`
- `variant`: `default`, `awq`, `fp8`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `text_only`, `encoder_parallel` (default: `encoder_parallel`)

## Install

Requires vLLM >= 0.10.2.

Docker image: `vllm/vllm-tpu:latest`

## Serve

```bash
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
  --tensor-parallel-size 1 \
  --mm-encoder-tp-mode data
```

Docker equivalent:

```bash
docker run --privileged --network host \
  -v /dev/shm:/dev/shm \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-tpu:latest Qwen/Qwen2.5-VL-7B-Instruct \
  --tensor-parallel-size 1 \
  --mm-encoder-tp-mode data
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Qwen2.5-VL-7B-Instruct is the small dense vision-language model in the Qwen2.5-VL series. At 7B it fits comfortably on a single accelerator — one TPU v6e (Trillium) chip, or a single 24GB+ GPU — and is also validated for Intel Xeon 6 CPU serving. BF16 is the precision used in training, so BF16 inference gives the best accuracy.

For the 72B sibling, see [Qwen2.5-VL-72B-Instruct](/Qwen/Qwen2.5-VL-72B-Instruct).

### Prerequisites

#### NVIDIA
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### Intel Xeon 6 CPUs

- Hardware: Intel Xeon 6 CPUs
- vLLM CPU >= 0.8.5 for this validated Xeon 6 configuration

For Intel and AMD x86 CPUs, follow the
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
installation instructions.

```bash
docker pull vllm/vllm-openai-cpu:latest-x86_64
```

#### TPU (Trillium / v6e)
Use the official vLLM TPU image. The 7B model fits on a single v6e chip (`v6e-1`, `--topology 1x1`).
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
docker run --rm --privileged --net=host \
  --shm-size=16G \
  -e HF_HOME -e HF_TOKEN \
  vllm/vllm-tpu:latest \
  vllm serve ...
```

### Deployment Configurations

#### Intel Xeon 6

Choose tensor parallelism based on the system topology; the recipe does not
prescribe a fixed TP/DP layout or hard-code NUMA node IDs.

```bash
vllm serve Qwen/Qwen2.5-VL-7B-Instruct
```

Docker:

```bash
docker run \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen2.5-VL-7B-Instruct \
  --tensor-parallel-size 1 \
  --mm-encoder-tp-mode data
```

##### Runtime and Platform Tuning

The following settings are intentionally not prescribed as portable CPU model
defaults because they depend on the workload, hardware, or runtime environment:

- `--max-num-batched-tokens`: scheduler/throughput tuning.
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
- `--gpu-memory-utilization`: platform memory-budget tuning.
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.

These settings may still appear in validated hardware-specific overrides. Tune
them at deployment time based on platform resources, workload shape, and
latency/throughput goals.

#### TPU v6e (Trillium, single chip)
Verified end-to-end on a single v6e chip.
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
  --tensor-parallel-size 1 \
  --pipeline-parallel-size 1 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.98 \
  --max-model-len 16384 \
  --limit-mm-per-prompt '{"image":10,"video":0}' \
  --mm-processor-kwargs '{"max_pixels":1003520}' \
  --guided-decoding-backend xgrammar \
  --disable-chunked-mm-input
```

#### Single GPU (BF16, TP=1)
```bash
export CUDA_VISIBLE_DEVICES=0
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --limit-mm-per-prompt '{"image":2,"video":0}'
```

#### Multi-GPU (DP=4)
For higher throughput across a node, data parallelism works better than TP for a model this size.
```bash
export CUDA_VISIBLE_DEVICES=0,1,2,3
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --data-parallel-size 4 \
  --limit-mm-per-prompt '{"image":2,"video":0}'
```

### Configuration Tips

- `--max-model-len 16384` is a safe default on a single v6e chip (native context is 128K); raise it if you have headroom.
- `--limit-mm-per-prompt` caps incoming multimodal requests per prompt.
- `--mm-processor-kwargs '{"max_pixels":1003520}'` bounds the per-image resolution to control encoder cost.
- vLLM uses 90% of device memory by default; on TPU the recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.
- `--disable-chunked-mm-input` is recommended on TPU for stable multimodal batching.

### Benchmarking

Launch the server with `--no-enable-prefix-caching` to get consistent measurements.

#### VisionArena-Chat
```bash
vllm bench serve \
  --host 0.0.0.0 \
  --port 8000 \
  --backend openai-chat \
  --endpoint /v1/chat/completions \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --dataset-name hf \
  --dataset-path lmarena-ai/VisionArena-Chat \
  --num-prompts 128
```

#### Random Synthetic
```bash
vllm bench serve \
  --host 0.0.0.0 \
  --port 8000 \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 128
```

### References

- [Model card](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct)
- [Qwen2.5-VL on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen2.5-VL)
- [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)