{ "hf_id": "Qwen/Qwen3-4B", "meta": { "title": "Qwen3-4B", "slug": "qwen3-4b", "provider": "Qwen", "description": "Qwen3 4B dense model with hybrid thinking/non-thinking modes — fits on a single TPU v6e chip, one GPU or one Xeon 6 NUMA node.", "date_added": "2025-10-15", "date_updated": "2026-09-12", "tasks": [ "text" ], "performance_headline": "Verified on TPU v6e (Trillium) with BF16 on a single chip", "related_recipes": [ "Qwen/Qwen3-32B" ], "hardware": { "trillium": "verified", "xeon6": "verified" } }, "recommended_command": { "hardware": "trillium", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-tpu:latest", "command": "vllm serve Qwen/Qwen3-4B \\\n --tensor-parallel-size 1 \\\n --enable-auto-tool-choice \\\n --tool-call-parser hermes \\\n --reasoning-parser qwen3", "argv": [ "vllm", "serve", "Qwen/Qwen3-4B", "--tensor-parallel-size", "1", "--enable-auto-tool-choice", "--tool-call-parser", "hermes", "--reasoning-parser", "qwen3" ], "docker_command": "docker run --privileged --network host \\\n -v /dev/shm:/dev/shm \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-tpu:latest Qwen/Qwen3-4B \\\n --tensor-parallel-size 1 \\\n --enable-auto-tool-choice \\\n --tool-call-parser hermes \\\n --reasoning-parser qwen3", "docker_argv": [ "docker", "run", "--privileged", "--network", "host", "-v", "/dev/shm:/dev/shm", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-tpu:latest", "Qwen/Qwen3-4B", "--tensor-parallel-size", "1", "--enable-auto-tool-choice", "--tool-call-parser", "hermes", "--reasoning-parser", "qwen3" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "Google", "generation": "tpu", "display_name": "TPU v6e (Trillium)", "description": "Google Cloud TPU v6e Trillium · 32 GB HBM/chip · served via vLLM TPU", "gpu_count": 8, "vram_gb": 256, "multi_node": false, "restricted": true }, "alternatives": { "multi_node_tp": "/Qwen/Qwen3-4B/strategies/multi_node_tp.json" }, "by_hardware": { "trillium": "/Qwen/Qwen3-4B/hw/trillium.json", "xeon6": "/Qwen/Qwen3-4B/hw/xeon6.json" } }, "model": { "model_id": "Qwen/Qwen3-4B", "min_vllm_version": "0.8.5", "architecture": "dense", "parameter_count": "4B", "active_parameters": "4B", "context_length": 40960, "base_args": [], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "tool_calling": { "description": "Enable automatic tool choice with the Hermes parser", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "hermes" ] }, "reasoning": { "description": "Enable Qwen3 thinking mode (chain-of-thought) with the Qwen3 reasoning parser", "args": [ "--reasoning-parser", "qwen3" ] }, "long_context": { "description": "YaRN RoPE scaling to the model card's validated 131,072-token context. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts — it can cost quality below 32K.", "args": [ "--hf-overrides", "{\"max_position_embeddings\":131072,\"rope_parameters\":{\"rope_type\":\"yarn\",\"factor\":4.0,\"original_max_position_embeddings\":32768}}", "--max-model-len", "131072" ] } }, "opt_in_features": [ "long_context" ], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 10, "description": "Full precision BF16 — fits on a single TPU v6e chip or one 16GB+ GPU or one Xeon 6 NUMA node", "supported_hardware": [ "trillium", "xeon6" ] }, "fp8": { "model_id": "Qwen/Qwen3-4B-FP8", "precision": "fp8", "vram_minimum_gb": 5, "description": "Qwen official FP8 checkpoint", "supported_hardware": [ "trillium" ], "json": "/Qwen/Qwen3-4B-FP8.json" }, "redhat_fp8": { "model_id": "RedHatAI/Qwen3-4B-FP8-dynamic", "precision": "fp8", "label": "FP8 Dynamic (RedHat)", "vram_minimum_gb": 5, "description": "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization", "supported_hardware": [ "trillium" ], "extra_args": [ "--kv-cache-dtype", "fp8" ], "json": "/RedHatAI/Qwen3-4B-FP8-dynamic.json" } }, "kv_offload_support": { "offloading_cpu": "verified", "offloading_fs": "verified" }, "hardware_overrides": {}, "guide": "## Overview\n\nQwen3-4B is the small dense model in the Qwen3 series, with hybrid thinking / non-thinking modes. At 4B it fits on a single accelerator — one TPU v6e (Trillium) chip, a single 16GB+ GPU or one Xeon 6 NUMA node — making it the cheapest entry point in the family. BF16 is the training precision and gives the best accuracy.\n\nToggle thinking mode per request with `enable_thinking` in `chat_template_kwargs`, or set the **Reasoning** feature to add the Qwen3 reasoning parser so `` content is split into a separate field.\n\n## Prerequisites\n\n### NVIDIA\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto\n```\n\n### pip (Intel Xeon 6 CPUs)\nFor Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.\n\n### TPU (Trillium / v6e)\nUse the official vLLM TPU image. The 4B model fits on a single v6e chip (`v6e-1`, `--topology 1x1`).\n```bash\nexport HF_HOME=/dev/shm\nexport HF_TOKEN=\n```\n\n## Deployment Configurations\n\n### TPU v6e (Trillium, single chip, TP=1)\nVerified end-to-end on a single v6e chip.\n```bash\nexport HF_HOME=/dev/shm\nexport HF_TOKEN=\nexport TP=1\nexport MAX_MODEL_LEN=4096\nvllm serve Qwen/Qwen3-4B \\\n --seed 42 \\\n --gpu-memory-utilization 0.98 \\\n --max-num-batched-tokens 1024 \\\n --max-num-seqs 128 \\\n --tensor-parallel-size $TP \\\n --max-model-len $MAX_MODEL_LEN\n```\n\n### Single GPU (BF16)\n```bash\nvllm serve Qwen/Qwen3-4B \\\n --host 0.0.0.0 \\\n --port 8000\n```\n\n### Intel Xeon 6 Deployment via Docker\n\nLaunch the x86 CPU vLLM Docker container for `Qwen/Qwen3-4B`:\n\n```bash\ndocker run \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-4B \\\n --tensor-parallel-size 1 \\\n --enable-auto-tool-choice \\\n --tool-call-parser hermes \\\n --reasoning-parser qwen3\n```\n\n## Configuration Tips\n\n- `--max-model-len 4096` is the TPU-recipe default; native context is 40K — raise it if you have headroom.\n- vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.\n- For thinking mode Qwen recommends `temperature=0.6, top_p=0.95, top_k=20`; for non-thinking, `temperature=0.7, top_p=0.8`.\n\n## Benchmarking\n\nLaunch the server with `--no-enable-prefix-caching` for consistent measurements.\n\n```bash\nvllm bench serve \\\n --host 0.0.0.0 \\\n --port 8000 \\\n --model Qwen/Qwen3-4B \\\n --dataset-name random \\\n --random-input-len 1024 \\\n --random-output-len 1024 \\\n --num-prompts 128\n```\n\n## References\n\n- [Model card](https://huggingface.co/Qwen/Qwen3-4B)\n- [Qwen3 on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/Qwen/Qwen3-4B.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/Qwen/Qwen3-4B.yaml", "overlays": [] } }