# meta-llama/Llama-3.2-1B — vLLM serving recipe

> Meta Llama 3.2 1.23B dense pretrained language model with 128K context, validated for BF16 serving on Intel Xeon 6.

Configuration: hardware `xeon6` (4× Xeon 6) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16)

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/meta-llama/Llama-3.2-1B
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/meta-llama/Llama-3.2-1B.json?hardware=xeon6

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `xeon6`
- `variant`: `default`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)

## Install

Requires vLLM >= 0.6.2.

Docker image: `vllm/vllm-openai-cpu:latest-x86_64`

## Serve

```bash
vllm serve meta-llama/Llama-3.2-1B \
  --tensor-parallel-size 1
```

Docker equivalent:

```bash
docker run --shm-size=16g \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest-x86_64 meta-llama/Llama-3.2-1B \
  --tensor-parallel-size 1
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Llama 3.2 1B is Meta's compact pretrained dense language model with 128K
context. This recipe includes an Intel Xeon 6 CPU configuration validated
with BF16 serving.

### Prerequisites

- Hardware: 1x Xeon 6 NUMA node
- vLLM >= 0.6.2

#### pip (Intel Xeon 6 CPUs)

For Intel and AMD x86 CPUs, follow the
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
installation instructions.

#### Docker (Intel Xeon 6 CPUs)

```bash
docker pull vllm/vllm-openai-cpu:latest-x86_64
```

### Runtime and Platform Tuning

The following validation settings are not model requirements and are intentionally
not portable CPU recipe defaults:

- `--max-num-batched-tokens` and `--max-num-seqs`: scheduler batch/concurrency
  tuning for workload shape, latency targets, and platform capacity.
- `--gpu-memory-utilization`: platform memory-budget tuning.
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: runtime operational timeout tuning.

Hardware-specific overrides may still use these settings when they are part of a
validated platform configuration.

### Intel Xeon 6

Choose the tensor parallel size based on the system topology and deployment
requirements. Add `--tensor-parallel-size <N>` when needed; the recipe does
not prescribe a Xeon 6 TP value or hard-code NUMA node IDs.

```bash
vllm serve meta-llama/Llama-3.2-1B
```

Docker:

```bash
docker run \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest-x86_64 meta-llama/Llama-3.2-1B \
  --tensor-parallel-size 1
```

### Client Usage

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.completions.create(
    model="meta-llama/Llama-3.2-1B",
    prompt="Explain tensor parallelism briefly.",
    max_tokens=128,
)
print(response.choices[0].text)
```

### References

- [Model card](https://huggingface.co/meta-llama/Llama-3.2-1B)
- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)