{
  "hf_id": "Qwen/QwQ-32B",
  "meta": {
    "title": "QwQ-32B",
    "slug": "qwq-32b",
    "provider": "Qwen",
    "description": "Qwen's 32B dense reasoning model, validated for Intel Xeon 6 CPU execution.",
    "date_added": "2026-08-17",
    "date_updated": "2026-09-12",
    "tasks": [
      "text"
    ],
    "performance_headline": "Validated for Intel Xeon 6 CPU execution",
    "related_recipes": [
      "Qwen/Qwen3-32B"
    ],
    "hardware": {
      "xeon6": "verified"
    }
  },
  "recommended_command": {
    "hardware": "xeon6",
    "strategy": "single_node_tp",
    "variant": "default",
    "node_count": 1,
    "deploy_type": "single_node",
    "env": {},
    "docker_image": "vllm/vllm-openai-cpu:latest-x86_64",
    "command": "vllm serve Qwen/QwQ-32B \\\n  --tensor-parallel-size 1 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser hermes \\\n  --reasoning-parser deepseek_r1",
    "argv": [
      "vllm",
      "serve",
      "Qwen/QwQ-32B",
      "--tensor-parallel-size",
      "1",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "hermes",
      "--reasoning-parser",
      "deepseek_r1"
    ],
    "docker_command": "docker run --shm-size=16g \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-openai-cpu:latest-x86_64 Qwen/QwQ-32B \\\n  --tensor-parallel-size 1 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser hermes \\\n  --reasoning-parser deepseek_r1",
    "docker_argv": [
      "docker",
      "run",
      "--shm-size",
      "16g",
      "--privileged",
      "--ipc=host",
      "-p",
      "8000:8000",
      "-v",
      "~/.cache/huggingface:/root/.cache/huggingface",
      "vllm/vllm-openai-cpu:latest-x86_64",
      "Qwen/QwQ-32B",
      "--tensor-parallel-size",
      "1",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "hermes",
      "--reasoning-parser",
      "deepseek_r1"
    ],
    "strategy_spec": {
      "name": "single_node_tp",
      "deploy_type": "single_node",
      "display_name": "Tensor Parallel",
      "short_label": "TP",
      "orientation": "latency",
      "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n",
      "hardware_match": {
        "min_gpus": 1,
        "max_gpus": 8,
        "multi_node": false
      },
      "vllm_args": [],
      "parallel_flag": "--tensor-parallel-size"
    },
    "hardware_profile": {
      "brand": "Intel",
      "generation": "cpu",
      "display_name": "Xeon 6",
      "description": "Intel Xeon 6 CPU NUMA node · x86 CPU backend",
      "gpu_count": 4,
      "vram_gb": 1024,
      "multi_node": false,
      "restricted": true
    },
    "alternatives": {
      "multi_node_tp": "/Qwen/QwQ-32B/strategies/multi_node_tp.json"
    },
    "by_hardware": {
      "xeon6": "/Qwen/QwQ-32B/hw/xeon6.json"
    }
  },
  "model": {
    "model_id": "Qwen/QwQ-32B",
    "min_vllm_version": "0.7.3",
    "architecture": "dense",
    "parameter_count": "32B",
    "active_parameters": "32B",
    "context_length": 40960,
    "base_args": [],
    "base_env": {},
    "install": {
      "pip": {
        "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto"
      },
      "docker": {
        "command": "docker pull vllm/vllm-openai:latest"
      }
    }
  },
  "features": {
    "tool_calling": {
      "description": "Enable automatic tool choice with the Hermes parser",
      "args": [
        "--enable-auto-tool-choice",
        "--tool-call-parser",
        "hermes"
      ]
    },
    "reasoning": {
      "description": "Extract QwQ reasoning content with the DeepSeek-R1 reasoning parser",
      "args": [
        "--reasoning-parser",
        "deepseek_r1"
      ]
    },
    "long_context": {
      "description": "YaRN RoPE scaling to the full 131,072-token context, which the model card requires for prompts over 8,192 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts.",
      "args": [
        "--hf-overrides",
        "{\"max_position_embeddings\":131072,\"rope_parameters\":{\"rope_type\":\"yarn\",\"factor\":4.0,\"original_max_position_embeddings\":32768}}",
        "--max-model-len",
        "131072"
      ]
    }
  },
  "opt_in_features": [
    "long_context"
  ],
  "variants": {
    "default": {
      "precision": "bf16",
      "vram_minimum_gb": 77,
      "description": "Full precision BF16",
      "supported_hardware": [
        "xeon6"
      ]
    },
    "w8a8_int8": {
      "model_id": "RedHatAI/QwQ-32B-quantized.w8a8",
      "precision": "int8",
      "vram_minimum_gb": 39,
      "description": "Red Hat AI INT8 W8A8 compressed-tensors checkpoint",
      "supported_hardware": [
        "xeon6"
      ],
      "json": "/RedHatAI/QwQ-32B-quantized.w8a8.json"
    },
    "awq": {
      "model_id": "Qwen/QwQ-32B-AWQ",
      "precision": "int4",
      "vram_minimum_gb": 20,
      "description": "Qwen official AWQ 4-bit checkpoint",
      "supported_hardware": [
        "xeon6"
      ],
      "extra_args": [
        "--quantization",
        "awq"
      ],
      "json": "/Qwen/QwQ-32B-AWQ.json"
    }
  },
  "kv_offload_support": {
    "offloading_cpu": "verified",
    "offloading_fs": "verified"
  },
  "hardware_overrides": {},
  "guide": "## Overview\n\nQwQ-32B is Qwen's 32B dense reasoning model. The CPU model validation table\nmarks it supported on Intel Xeon 6.\n\n## Prerequisites\n\n- Hardware: Intel Xeon 6 CPUs\n- vLLM CPU >= 0.8.5 for this validated Xeon 6 configuration\n\n### pip (Intel Xeon 6 CPUs)\n\nFor Intel and AMD x86 CPUs, follow the\n[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)\ninstallation instructions.\n\n### Docker (Intel Xeon 6 CPUs)\n\n```bash\ndocker pull vllm/vllm-openai-cpu:latest-x86_64\n```\n\n## Intel Xeon 6\n\n```bash\nvllm serve Qwen/QwQ-32B \\\n  --tensor-parallel-size 1 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser hermes \\\n  --reasoning-parser deepseek_r1\n```\n\nDocker (the image entrypoint is `vllm serve`):\n\n```bash\ndocker run \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-openai-cpu:latest-x86_64 Qwen/QwQ-32B \\\n  --tensor-parallel-size 1 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser hermes \\\n  --reasoning-parser deepseek_r1\n```\n\nW8A8 INT8:\n\n```bash\nvllm serve RedHatAI/QwQ-32B-quantized.w8a8\n```\n\nThe example uses TP=1. Adjust tensor parallelism based on system topology;\ntopology-specific CPU binding is intentionally not hard-coded.\n\n## Runtime and Platform Tuning\n\nThe following settings are intentionally not prescribed as portable CPU model\ndefaults because they depend on the workload, hardware, or runtime environment:\n\n- `--max-num-batched-tokens`: scheduler/throughput tuning.\n- `--max-num-seqs`: concurrency and scheduler-capacity tuning.\n- `--gpu-memory-utilization`: platform memory-budget tuning.\n- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.\n- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.\n\nThese settings may still appear in validated hardware-specific overrides. Tune\nthem at deployment time based on platform resources, workload shape, and\nlatency/throughput goals.\n\n## References\n\n- [Model card](https://huggingface.co/Qwen/QwQ-32B)\n- [W8A8 variant](https://huggingface.co/RedHatAI/QwQ-32B-quantized.w8a8)\n- [vLLM reasoning outputs](https://docs.vllm.ai/en/latest/features/reasoning_outputs.html)\n- [vLLM tool calling](https://docs.vllm.ai/en/latest/features/tool_calling.html)\n",
  "source": {
    "source": "vllm",
    "repo": "vllm-project/vllm",
    "ref": "main",
    "sha": "389ffebe555705c85337025d5df108d0384d365e",
    "path": "recipes/models/Qwen/QwQ-32B.yaml",
    "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/Qwen/QwQ-32B.yaml",
    "overlays": []
  }
}