# meta-llama/Llama-3.1-8B — vLLM serving recipe

> Meta Llama 3.1 8B dense base language model with a CPU-validated Red Hat AI W8A8 variant.

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16)

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/meta-llama/Llama-3.1-8B
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/meta-llama/Llama-3.1-8B.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`, `xeon6`
- `variant`: `default`, `w8a8_int8`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)

## Install

Requires vLLM >= 0.6.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve meta-llama/Llama-3.1-8B \
  --tensor-parallel-size 1
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest meta-llama/Llama-3.1-8B \
  --tensor-parallel-size 1
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Llama 3.1 8B is Meta's dense pretrained base model with 128K context. This
recipe is separate from Llama-3.1-8B-Instruct because the Red Hat AI
`Meta-Llama-3.1-8B-quantized.w8a8` checkpoint is quantized from the base
model, not the instruction-tuned checkpoint.

The recipe retains portable model-level CPU settings while leaving TP/DP and
host-specific CPU placement as deployment choices.

### Prerequisites

- Hardware: 2x Xeon 6 NUMA nodes
- vLLM >= 0.6.0

#### pip (Intel Xeon 6 CPUs)

For Intel and AMD x86 CPUs, follow the
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
installation instructions.

#### Docker (Intel Xeon 6 CPUs)

```bash
docker pull vllm/vllm-openai-cpu:latest-x86_64
```

### Intel Xeon 6

```bash
vllm serve RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8
```

Docker:

```bash
docker run \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest-x86_64 RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8 \
  --tensor-parallel-size 2
```

### Runtime and Platform Tuning

The following validation settings are not model requirements and are intentionally
not portable CPU recipe defaults:

- `--max-num-batched-tokens` and `--max-num-seqs`: scheduler batch/concurrency
  tuning for workload shape, latency targets, and platform capacity.
- `--gpu-memory-utilization`: platform memory-budget tuning.
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: runtime operational timeout tuning.

Hardware-specific overrides may still use these settings when they are part of a
validated platform configuration.

### Configuration Notes

- The validation TP/DP layout is benchmark topology and is not hard-coded.
- Llama has native vLLM architecture support, so `--trust-remote-code` is not required by this recipe.

### References

- [Parent model](https://huggingface.co/meta-llama/Llama-3.1-8B)
- [Red Hat AI W8A8 variant](https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8)
- [Llama-3.1-8B-Instruct recipe](https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct)
- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)