# meta-llama/Llama-3.2-1B-Instruct — vLLM serving recipe

> Meta Llama 3.2 1B Instruct with CPU-validated Red Hat AI W8A8 and AMead10 AWQ variants.

Configuration: hardware `xeon6` (4× Xeon 6) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16)

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/meta-llama/Llama-3.2-1B-Instruct
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/meta-llama/Llama-3.2-1B-Instruct.json?hardware=xeon6

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `xeon6`
- `variant`: `default`, `w8a8_int8`, `awq_int4`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)

## Install

Requires vLLM >= 0.6.2.

Docker image: `vllm/vllm-openai-cpu:latest-x86_64`

## Serve

```bash
vllm serve meta-llama/Llama-3.2-1B-Instruct \
  --tensor-parallel-size 1
```

Docker equivalent:

```bash
docker run --shm-size=16g \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest-x86_64 meta-llama/Llama-3.2-1B-Instruct \
  --tensor-parallel-size 1
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

This recipe groups the Llama 3.2 1B Instruct family under the canonical Meta
parent. The Xeon 6 validation workbook has PASS rows for the Red Hat AI W8A8
and AMead10 AWQ variants.

The W8A8 checkpoint is detected from compressed-tensors metadata. The AWQ
variant uses the recipe's `--quantization awq` selector.

### Prerequisites

- Hardware: 1x Xeon6/Xeon5 NUMA node
- vLLM >= 0.6.2

#### pip (Intel Xeon 6 CPUs)

For Intel and AMD x86 CPUs, follow the
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
installation instructions.

#### Docker (Intel Xeon 6 CPUs)

```bash
docker pull vllm/vllm-openai-cpu:latest-x86_64
```

### Intel Xeon 6

```bash
vllm serve meta-llama/Llama-3.2-1B-Instruct \
  --tensor-parallel-size 1
```

Docker (the image entrypoint is `vllm serve`):

```bash
docker run \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest-x86_64 meta-llama/Llama-3.2-1B-Instruct \
  --tensor-parallel-size 1
```

The example uses TP=1. Adjust TP/DP for the deployment topology;
host-specific CPU binding is intentionally not hard-coded.

### Runtime and Platform Tuning

The following validation settings are not model requirements and are intentionally
not portable CPU recipe defaults:

- `--max-num-batched-tokens` and `--max-num-seqs`: scheduler batch/concurrency
  tuning for workload shape, latency targets, and platform capacity.
- `--gpu-memory-utilization`: platform memory-budget tuning.
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: runtime operational timeout tuning.

Hardware-specific overrides may still use these settings when they are part of a
validated platform configuration.

### References

- [Meta parent model](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct)
- [Red Hat AI W8A8 checkpoint](https://huggingface.co/RedHatAI/Llama-3.2-1B-Instruct-quantized.w8a8)
- [AMead10 AWQ checkpoint](https://huggingface.co/AMead10/Llama-3.2-1B-Instruct-AWQ)