{ "hf_id": "Qwen/Qwen2.5-32B", "meta": { "title": "Qwen2.5-32B", "slug": "qwen2.5-32b", "provider": "Qwen", "description": "Qwen2.5 32B dense base (pretrained) language model for text completion — verified on TPU v6e (Trillium).", "date_added": "2026-06-05", "date_updated": "2026-06-04", "tasks": [ "text" ], "performance_headline": "Verified on TPU v6e (Trillium) with BF16, TP=4 on a 2x2 slice", "related_recipes": [], "hardware": { "trillium": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve Qwen/Qwen2.5-32B \\\n --tensor-parallel-size 1", "argv": [ "vllm", "serve", "Qwen/Qwen2.5-32B", "--tensor-parallel-size", "1" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest Qwen/Qwen2.5-32B \\\n --tensor-parallel-size 1", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "Qwen/Qwen2.5-32B", "--tensor-parallel-size", "1" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "multi_node_tp": "/Qwen/Qwen2.5-32B/strategies/multi_node_tp.json", "kv_store_centralized_mooncake": "/Qwen/Qwen2.5-32B/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/Qwen/Qwen2.5-32B/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/Qwen/Qwen2.5-32B/hw/h200.json", "h100": "/Qwen/Qwen2.5-32B/hw/h100.json", "b200": "/Qwen/Qwen2.5-32B/hw/b200.json", "gb200": "/Qwen/Qwen2.5-32B/hw/gb200.json", "b300": "/Qwen/Qwen2.5-32B/hw/b300.json", "gb300": "/Qwen/Qwen2.5-32B/hw/gb300.json", "mi300x": "/Qwen/Qwen2.5-32B/hw/mi300x.json", "mi325x": "/Qwen/Qwen2.5-32B/hw/mi325x.json", "mi350x": "/Qwen/Qwen2.5-32B/hw/mi350x.json", "mi355x": "/Qwen/Qwen2.5-32B/hw/mi355x.json", "trillium": "/Qwen/Qwen2.5-32B/hw/trillium.json" } }, "model": { "model_id": "Qwen/Qwen2.5-32B", "min_vllm_version": "0.6.2", "architecture": "dense", "parameter_count": "32B", "active_parameters": "32B", "context_length": 131072, "base_args": [], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": {}, "opt_in_features": [], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 77, "description": "Full precision BF16 — TPU v6e 2x2 (4 chips, TP=4) or a single 80GB+ GPU" }, "fp8": { "model_id": "RedHatAI/Qwen2.5-32B-FP8-dynamic", "precision": "fp8", "vram_minimum_gb": 39, "description": "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization", "extra_args": [ "--kv-cache-dtype", "fp8" ], "json": "/RedHatAI/Qwen2.5-32B-FP8-dynamic.json" } }, "kv_offload_support": { "offloading_cpu": "verified", "offloading_fs": "verified" }, "hardware_overrides": {}, "guide": "## Overview\n\nQwen2.5-32B is the 32B dense **base** (pretrained, non-instruct) model in the Qwen2.5 series. It's a text-completion model — use the `/v1/completions` endpoint rather than chat, or pick `Qwen/Qwen2.5-32B-Instruct` if you need a chat-tuned variant. BF16 is the training precision and gives the best accuracy.\n\n## Prerequisites\n\n### NVIDIA\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto\n```\n\n### TPU (Trillium / v6e)\nUse the official vLLM TPU image. The 32B model is served on a `v6e-4` slice (`--topology 2x2`, 4 chips, TP=4).\n```bash\nexport HF_HOME=/dev/shm\nexport HF_TOKEN=\n```\n\n## Deployment Configurations\n\n### TPU v6e (Trillium, 2x2 slice, TP=4)\nVerified end-to-end on a 4-chip v6e slice.\n```bash\nexport HF_HOME=/dev/shm\nexport HF_TOKEN=\nexport TP=4\nexport MAX_MODEL_LEN=4096\nvllm serve Qwen/Qwen2.5-32B \\\n --seed 42 \\\n --gpu-memory-utilization 0.98 \\\n --max-num-batched-tokens 2048 \\\n --max-num-seqs 128 \\\n --tensor-parallel-size $TP \\\n --max-model-len $MAX_MODEL_LEN\n```\n\n### Single GPU / node (BF16)\n```bash\nvllm serve Qwen/Qwen2.5-32B \\\n --host 0.0.0.0 \\\n --port 8000 \\\n --tensor-parallel-size 2\n```\n\n## Configuration Tips\n\n- `--max-model-len 4096` keeps memory in check on a 4-chip v6e slice; native context is 128K — raise it if you have headroom.\n- vLLM has been V1-engine-only for a long time, so no `VLLM_USE_V1=1` export is needed even though the upstream TPU recipe still shows it.\n- vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.\n- `--max-num-batched-tokens` and `--max-num-seqs` tune batch throughput vs latency.\n\n## Benchmarking\n\nLaunch the server with `--no-enable-prefix-caching` for consistent measurements.\n\n```bash\nvllm bench serve \\\n --host 0.0.0.0 \\\n --port 8000 \\\n --model Qwen/Qwen2.5-32B \\\n --dataset-name random \\\n --random-input-len 1024 \\\n --random-output-len 1024 \\\n --num-prompts 128\n```\n\n## References\n\n- [Model card](https://huggingface.co/Qwen/Qwen2.5-32B)\n- [Qwen2.5-32B on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen2.5-32B)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/Qwen/Qwen2.5-32B.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/Qwen/Qwen2.5-32B.yaml", "overlays": [] } }