{
  "hf_id": "Qwen/Qwen2.5-VL-7B-Instruct",
  "meta": {
    "title": "Qwen2.5-VL-7B-Instruct",
    "slug": "qwen2.5-vl-7b-instruct",
    "provider": "Qwen",
    "description": "Qwen2.5-VL dense vision-language model (7B) for image and video understanding — fits on a single TPU v6e chip, one GPU, or Intel Xeon 6 CPU.",
    "date_added": "2025-08-22",
    "date_updated": "2026-09-12",
    "tasks": [
      "multimodal",
      "text"
    ],
    "performance_headline": "Verified on TPU v6e (Trillium) and Intel Xeon 6 CPU with BF16",
    "related_recipes": [
      "Qwen/Qwen2.5-VL-72B-Instruct"
    ],
    "hardware": {
      "trillium": "verified",
      "xeon6": "verified"
    }
  },
  "recommended_command": {
    "hardware": "trillium",
    "strategy": "single_node_tp",
    "variant": "default",
    "node_count": 1,
    "deploy_type": "single_node",
    "env": {},
    "docker_image": "vllm/vllm-tpu:latest",
    "command": "vllm serve Qwen/Qwen2.5-VL-7B-Instruct \\\n  --tensor-parallel-size 1 \\\n  --mm-encoder-tp-mode data",
    "argv": [
      "vllm",
      "serve",
      "Qwen/Qwen2.5-VL-7B-Instruct",
      "--tensor-parallel-size",
      "1",
      "--mm-encoder-tp-mode",
      "data"
    ],
    "docker_command": "docker run --privileged --network host \\\n  -v /dev/shm:/dev/shm \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-tpu:latest Qwen/Qwen2.5-VL-7B-Instruct \\\n  --tensor-parallel-size 1 \\\n  --mm-encoder-tp-mode data",
    "docker_argv": [
      "docker",
      "run",
      "--privileged",
      "--network",
      "host",
      "-v",
      "/dev/shm:/dev/shm",
      "--privileged",
      "--ipc=host",
      "-p",
      "8000:8000",
      "-v",
      "~/.cache/huggingface:/root/.cache/huggingface",
      "vllm/vllm-tpu:latest",
      "Qwen/Qwen2.5-VL-7B-Instruct",
      "--tensor-parallel-size",
      "1",
      "--mm-encoder-tp-mode",
      "data"
    ],
    "strategy_spec": {
      "name": "single_node_tp",
      "deploy_type": "single_node",
      "display_name": "Tensor Parallel",
      "short_label": "TP",
      "orientation": "latency",
      "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n",
      "hardware_match": {
        "min_gpus": 1,
        "max_gpus": 8,
        "multi_node": false
      },
      "vllm_args": [],
      "parallel_flag": "--tensor-parallel-size"
    },
    "hardware_profile": {
      "brand": "Google",
      "generation": "tpu",
      "display_name": "TPU v6e (Trillium)",
      "description": "Google Cloud TPU v6e Trillium · 32 GB HBM/chip · served via vLLM TPU",
      "gpu_count": 8,
      "vram_gb": 256,
      "multi_node": false,
      "restricted": true
    },
    "alternatives": {
      "multi_node_tp": "/Qwen/Qwen2.5-VL-7B-Instruct/strategies/multi_node_tp.json"
    },
    "by_hardware": {
      "trillium": "/Qwen/Qwen2.5-VL-7B-Instruct/hw/trillium.json",
      "xeon6": "/Qwen/Qwen2.5-VL-7B-Instruct/hw/xeon6.json"
    }
  },
  "model": {
    "model_id": "Qwen/Qwen2.5-VL-7B-Instruct",
    "min_vllm_version": "0.10.2",
    "architecture": "dense",
    "parameter_count": "7B",
    "active_parameters": "7B",
    "context_length": 128000,
    "base_args": [],
    "base_env": {},
    "install": {
      "pip": {
        "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto"
      },
      "docker": {
        "command": "docker pull vllm/vllm-openai:latest"
      }
    }
  },
  "features": {
    "text_only": {
      "description": "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel.",
      "args": [
        "--language-model-only"
      ]
    },
    "encoder_parallel": {
      "description": "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only.",
      "args": [
        "--mm-encoder-tp-mode",
        "data"
      ]
    }
  },
  "opt_in_features": [
    "text_only"
  ],
  "variants": {
    "default": {
      "precision": "bf16",
      "vram_minimum_gb": 17,
      "description": "Full precision BF16 — fits on a single TPU v6e chip or one 24GB+ GPU",
      "supported_hardware": [
        "trillium",
        "xeon6"
      ]
    },
    "awq": {
      "model_id": "Qwen/Qwen2.5-VL-7B-Instruct-AWQ",
      "precision": "int4",
      "vram_minimum_gb": 5,
      "description": "AWQ 4-bit quantized weights",
      "supported_hardware": [
        "trillium"
      ],
      "extra_args": [
        "--quantization",
        "awq"
      ],
      "json": "/Qwen/Qwen2.5-VL-7B-Instruct-AWQ.json"
    },
    "fp8": {
      "model_id": "RedHatAI/Qwen2.5-VL-7B-Instruct-FP8-Dynamic",
      "precision": "fp8",
      "vram_minimum_gb": 9,
      "description": "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization",
      "supported_hardware": [
        "trillium"
      ],
      "extra_args": [
        "--kv-cache-dtype",
        "fp8"
      ],
      "json": "/RedHatAI/Qwen2.5-VL-7B-Instruct-FP8-Dynamic.json"
    }
  },
  "kv_offload_support": {
    "offloading_cpu": "verified",
    "offloading_fs": "verified"
  },
  "hardware_overrides": {},
  "guide": "## Overview\n\nQwen2.5-VL-7B-Instruct is the small dense vision-language model in the Qwen2.5-VL series. At 7B it fits comfortably on a single accelerator — one TPU v6e (Trillium) chip, or a single 24GB+ GPU — and is also validated for Intel Xeon 6 CPU serving. BF16 is the precision used in training, so BF16 inference gives the best accuracy.\n\nFor the 72B sibling, see [Qwen2.5-VL-72B-Instruct](/Qwen/Qwen2.5-VL-72B-Instruct).\n\n## Prerequisites\n\n### NVIDIA\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto\n```\n\n### Intel Xeon 6 CPUs\n\n- Hardware: Intel Xeon 6 CPUs\n- vLLM CPU >= 0.8.5 for this validated Xeon 6 configuration\n\nFor Intel and AMD x86 CPUs, follow the\n[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)\ninstallation instructions.\n\n```bash\ndocker pull vllm/vllm-openai-cpu:latest-x86_64\n```\n\n### TPU (Trillium / v6e)\nUse the official vLLM TPU image. The 7B model fits on a single v6e chip (`v6e-1`, `--topology 1x1`).\n```bash\nexport HF_HOME=/dev/shm\nexport HF_TOKEN=<your HF token>\ndocker run --rm --privileged --net=host \\\n  --shm-size=16G \\\n  -e HF_HOME -e HF_TOKEN \\\n  vllm/vllm-tpu:latest \\\n  vllm serve ...\n```\n\n## Deployment Configurations\n\n### Intel Xeon 6\n\nChoose tensor parallelism based on the system topology; the recipe does not\nprescribe a fixed TP/DP layout or hard-code NUMA node IDs.\n\n```bash\nvllm serve Qwen/Qwen2.5-VL-7B-Instruct\n```\n\nDocker:\n\n```bash\ndocker run \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen2.5-VL-7B-Instruct \\\n  --tensor-parallel-size 1 \\\n  --mm-encoder-tp-mode data\n```\n\n#### Runtime and Platform Tuning\n\nThe following settings are intentionally not prescribed as portable CPU model\ndefaults because they depend on the workload, hardware, or runtime environment:\n\n- `--max-num-batched-tokens`: scheduler/throughput tuning.\n- `--max-num-seqs`: concurrency and scheduler-capacity tuning.\n- `--gpu-memory-utilization`: platform memory-budget tuning.\n- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.\n- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.\n\nThese settings may still appear in validated hardware-specific overrides. Tune\nthem at deployment time based on platform resources, workload shape, and\nlatency/throughput goals.\n\n### TPU v6e (Trillium, single chip)\nVerified end-to-end on a single v6e chip.\n```bash\nexport HF_HOME=/dev/shm\nexport HF_TOKEN=<your HF token>\nvllm serve Qwen/Qwen2.5-VL-7B-Instruct \\\n  --tensor-parallel-size 1 \\\n  --pipeline-parallel-size 1 \\\n  --dtype bfloat16 \\\n  --gpu-memory-utilization 0.98 \\\n  --max-model-len 16384 \\\n  --limit-mm-per-prompt '{\"image\":10,\"video\":0}' \\\n  --mm-processor-kwargs '{\"max_pixels\":1003520}' \\\n  --guided-decoding-backend xgrammar \\\n  --disable-chunked-mm-input\n```\n\n### Single GPU (BF16, TP=1)\n```bash\nexport CUDA_VISIBLE_DEVICES=0\nvllm serve Qwen/Qwen2.5-VL-7B-Instruct \\\n  --host 0.0.0.0 \\\n  --port 8000 \\\n  --limit-mm-per-prompt '{\"image\":2,\"video\":0}'\n```\n\n### Multi-GPU (DP=4)\nFor higher throughput across a node, data parallelism works better than TP for a model this size.\n```bash\nexport CUDA_VISIBLE_DEVICES=0,1,2,3\nvllm serve Qwen/Qwen2.5-VL-7B-Instruct \\\n  --host 0.0.0.0 \\\n  --port 8000 \\\n  --data-parallel-size 4 \\\n  --limit-mm-per-prompt '{\"image\":2,\"video\":0}'\n```\n\n## Configuration Tips\n\n- `--max-model-len 16384` is a safe default on a single v6e chip (native context is 128K); raise it if you have headroom.\n- `--limit-mm-per-prompt` caps incoming multimodal requests per prompt.\n- `--mm-processor-kwargs '{\"max_pixels\":1003520}'` bounds the per-image resolution to control encoder cost.\n- vLLM uses 90% of device memory by default; on TPU the recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.\n- `--disable-chunked-mm-input` is recommended on TPU for stable multimodal batching.\n\n## Benchmarking\n\nLaunch the server with `--no-enable-prefix-caching` to get consistent measurements.\n\n### VisionArena-Chat\n```bash\nvllm bench serve \\\n  --host 0.0.0.0 \\\n  --port 8000 \\\n  --backend openai-chat \\\n  --endpoint /v1/chat/completions \\\n  --model Qwen/Qwen2.5-VL-7B-Instruct \\\n  --dataset-name hf \\\n  --dataset-path lmarena-ai/VisionArena-Chat \\\n  --num-prompts 128\n```\n\n### Random Synthetic\n```bash\nvllm bench serve \\\n  --host 0.0.0.0 \\\n  --port 8000 \\\n  --model Qwen/Qwen2.5-VL-7B-Instruct \\\n  --dataset-name random \\\n  --random-input-len 8000 \\\n  --random-output-len 1000 \\\n  --num-prompts 128\n```\n\n## References\n\n- [Model card](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct)\n- [Qwen2.5-VL on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen2.5-VL)\n- [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)\n",
  "source": {
    "source": "vllm",
    "repo": "vllm-project/vllm",
    "ref": "main",
    "sha": "389ffebe555705c85337025d5df108d0384d365e",
    "path": "recipes/models/Qwen/Qwen2.5-VL-7B-Instruct.yaml",
    "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/Qwen/Qwen2.5-VL-7B-Instruct.yaml",
    "overlays": []
  }
}