{ "hf_id": "meta-llama/Llama-3.1-8B", "meta": { "title": "Llama-3.1-8B", "slug": "llama-3.1-8b", "provider": "Meta", "description": "Meta Llama 3.1 8B dense base language model with a CPU-validated Red Hat AI W8A8 variant.", "date_added": "2026-08-19", "date_updated": "2026-09-12", "tasks": [ "text" ], "performance_headline": "W8A8 variant validated on Intel Xeon 6", "related_recipes": [ "meta-llama/Llama-3.1-8B-Instruct" ], "hardware": { "xeon6": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve meta-llama/Llama-3.1-8B \\\n --tensor-parallel-size 1", "argv": [ "vllm", "serve", "meta-llama/Llama-3.1-8B", "--tensor-parallel-size", "1" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest meta-llama/Llama-3.1-8B \\\n --tensor-parallel-size 1", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "meta-llama/Llama-3.1-8B", "--tensor-parallel-size", "1" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "multi_node_tp": "/meta-llama/Llama-3.1-8B/strategies/multi_node_tp.json", "kv_store_centralized_mooncake": "/meta-llama/Llama-3.1-8B/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/meta-llama/Llama-3.1-8B/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/meta-llama/Llama-3.1-8B/hw/h200.json", "h100": "/meta-llama/Llama-3.1-8B/hw/h100.json", "b200": "/meta-llama/Llama-3.1-8B/hw/b200.json", "gb200": "/meta-llama/Llama-3.1-8B/hw/gb200.json", "b300": "/meta-llama/Llama-3.1-8B/hw/b300.json", "gb300": "/meta-llama/Llama-3.1-8B/hw/gb300.json", "mi300x": "/meta-llama/Llama-3.1-8B/hw/mi300x.json", "mi325x": "/meta-llama/Llama-3.1-8B/hw/mi325x.json", "mi350x": "/meta-llama/Llama-3.1-8B/hw/mi350x.json", "mi355x": "/meta-llama/Llama-3.1-8B/hw/mi355x.json", "xeon6": "/meta-llama/Llama-3.1-8B/hw/xeon6.json" } }, "model": { "model_id": "meta-llama/Llama-3.1-8B", "min_vllm_version": "0.6.0", "architecture": "dense", "parameter_count": "8B", "active_parameters": "8B", "context_length": 131072, "base_args": [], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": {}, "opt_in_features": [], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 20, "description": "Full precision BF16 parent checkpoint" }, "w8a8_int8": { "model_id": "RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8", "precision": "int8", "vram_minimum_gb": 10, "description": "Red Hat AI INT8 W8A8 compressed-tensors checkpoint", "supported_hardware": [ "xeon6" ], "json": "/RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8.json" } }, "hardware_overrides": {}, "guide": "## Overview\n\nLlama 3.1 8B is Meta's dense pretrained base model with 128K context. This\nrecipe is separate from Llama-3.1-8B-Instruct because the Red Hat AI\n`Meta-Llama-3.1-8B-quantized.w8a8` checkpoint is quantized from the base\nmodel, not the instruction-tuned checkpoint.\n\nThe recipe retains portable model-level CPU settings while leaving TP/DP and\nhost-specific CPU placement as deployment choices.\n\n## Prerequisites\n\n- Hardware: 2x Xeon 6 NUMA nodes\n- vLLM >= 0.6.0\n\n### pip (Intel Xeon 6 CPUs)\n\nFor Intel and AMD x86 CPUs, follow the\n[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)\ninstallation instructions.\n\n### Docker (Intel Xeon 6 CPUs)\n\n```bash\ndocker pull vllm/vllm-openai-cpu:latest-x86_64\n```\n\n## Intel Xeon 6\n\n```bash\nvllm serve RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8\n```\n\nDocker:\n\n```bash\ndocker run \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai-cpu:latest-x86_64 RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8 \\\n --tensor-parallel-size 2\n```\n\n## Runtime and Platform Tuning\n\nThe following validation settings are not model requirements and are intentionally\nnot portable CPU recipe defaults:\n\n- `--max-num-batched-tokens` and `--max-num-seqs`: scheduler batch/concurrency\n tuning for workload shape, latency targets, and platform capacity.\n- `--gpu-memory-utilization`: platform memory-budget tuning.\n- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.\n- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: runtime operational timeout tuning.\n\nHardware-specific overrides may still use these settings when they are part of a\nvalidated platform configuration.\n\n## Configuration Notes\n\n- The validation TP/DP layout is benchmark topology and is not hard-coded.\n- Llama has native vLLM architecture support, so `--trust-remote-code` is not required by this recipe.\n\n## References\n\n- [Parent model](https://huggingface.co/meta-llama/Llama-3.1-8B)\n- [Red Hat AI W8A8 variant](https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8)\n- [Llama-3.1-8B-Instruct recipe](https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct)\n- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/meta-llama/Llama-3.1-8B.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/meta-llama/Llama-3.1-8B.yaml", "overlays": [] } }