# Qwen/Qwen3-4B — vLLM serving recipe

> Qwen3 4B dense model with hybrid thinking/non-thinking modes — fits on a single TPU v6e chip, one GPU or one Xeon 6 NUMA node.

Configuration: hardware `trillium` (8× TPU v6e (Trillium)) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/Qwen/Qwen3-4B
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/Qwen/Qwen3-4B.json?hardware=trillium

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `trillium`, `xeon6`
- `variant`: `default`, `fp8`, `redhat_fp8`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `long_context` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.8.5.

Docker image: `vllm/vllm-tpu:latest`

## Serve

```bash
vllm serve Qwen/Qwen3-4B \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --reasoning-parser qwen3
```

Docker equivalent:

```bash
docker run --privileged --network host \
  -v /dev/shm:/dev/shm \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-tpu:latest Qwen/Qwen3-4B \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --reasoning-parser qwen3
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Qwen3-4B is the small dense model in the Qwen3 series, with hybrid thinking / non-thinking modes. At 4B it fits on a single accelerator — one TPU v6e (Trillium) chip, a single 16GB+ GPU or one Xeon 6 NUMA node — making it the cheapest entry point in the family. BF16 is the training precision and gives the best accuracy.

Toggle thinking mode per request with `enable_thinking` in `chat_template_kwargs`, or set the **Reasoning** feature to add the Qwen3 reasoning parser so `<think>` content is split into a separate field.

### Prerequisites

#### NVIDIA
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### pip (Intel Xeon 6 CPUs)
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.

#### TPU (Trillium / v6e)
Use the official vLLM TPU image. The 4B model fits on a single v6e chip (`v6e-1`, `--topology 1x1`).
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
```

### Deployment Configurations

#### TPU v6e (Trillium, single chip, TP=1)
Verified end-to-end on a single v6e chip.
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
export TP=1
export MAX_MODEL_LEN=4096
vllm serve Qwen/Qwen3-4B \
  --seed 42 \
  --gpu-memory-utilization 0.98 \
  --max-num-batched-tokens 1024 \
  --max-num-seqs 128 \
  --tensor-parallel-size $TP \
  --max-model-len $MAX_MODEL_LEN
```

#### Single GPU (BF16)
```bash
vllm serve Qwen/Qwen3-4B \
  --host 0.0.0.0 \
  --port 8000
```

#### Intel Xeon 6 Deployment via Docker

Launch the x86 CPU vLLM Docker container for `Qwen/Qwen3-4B`:

```bash
docker run \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-4B \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --reasoning-parser qwen3
```

### Configuration Tips

- `--max-model-len 4096` is the TPU-recipe default; native context is 40K — raise it if you have headroom.
- vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.
- For thinking mode Qwen recommends `temperature=0.6, top_p=0.95, top_k=20`; for non-thinking, `temperature=0.7, top_p=0.8`.

### Benchmarking

Launch the server with `--no-enable-prefix-caching` for consistent measurements.

```bash
vllm bench serve \
  --host 0.0.0.0 \
  --port 8000 \
  --model Qwen/Qwen3-4B \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 128
```

### References

- [Model card](https://huggingface.co/Qwen/Qwen3-4B)
- [Qwen3 on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3)