{
  "hf_id": "meta-llama/Llama-3.2-1B",
  "meta": {
    "title": "Llama-3.2-1B",
    "slug": "llama-3.2-1b",
    "provider": "Meta",
    "description": "Meta Llama 3.2 1.23B dense pretrained language model with 128K context, validated for BF16 serving on Intel Xeon 6.",
    "date_added": "2026-08-18",
    "date_updated": "2026-09-12",
    "tasks": [
      "text"
    ],
    "performance_headline": "Validated on Intel Xeon 6 (GNR) with BF16 serving",
    "related_recipes": [
      "meta-llama/Llama-3.2-3B-Instruct",
      "meta-llama/Llama-3.1-8B-Instruct"
    ],
    "hardware": {
      "xeon6": "verified"
    }
  },
  "recommended_command": {
    "hardware": "xeon6",
    "strategy": "single_node_tp",
    "variant": "default",
    "node_count": 1,
    "deploy_type": "single_node",
    "env": {},
    "docker_image": "vllm/vllm-openai-cpu:latest-x86_64",
    "command": "vllm serve meta-llama/Llama-3.2-1B \\\n  --tensor-parallel-size 1",
    "argv": [
      "vllm",
      "serve",
      "meta-llama/Llama-3.2-1B",
      "--tensor-parallel-size",
      "1"
    ],
    "docker_command": "docker run --shm-size=16g \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-openai-cpu:latest-x86_64 meta-llama/Llama-3.2-1B \\\n  --tensor-parallel-size 1",
    "docker_argv": [
      "docker",
      "run",
      "--shm-size",
      "16g",
      "--privileged",
      "--ipc=host",
      "-p",
      "8000:8000",
      "-v",
      "~/.cache/huggingface:/root/.cache/huggingface",
      "vllm/vllm-openai-cpu:latest-x86_64",
      "meta-llama/Llama-3.2-1B",
      "--tensor-parallel-size",
      "1"
    ],
    "strategy_spec": {
      "name": "single_node_tp",
      "deploy_type": "single_node",
      "display_name": "Tensor Parallel",
      "short_label": "TP",
      "orientation": "latency",
      "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n",
      "hardware_match": {
        "min_gpus": 1,
        "max_gpus": 8,
        "multi_node": false
      },
      "vllm_args": [],
      "parallel_flag": "--tensor-parallel-size"
    },
    "hardware_profile": {
      "brand": "Intel",
      "generation": "cpu",
      "display_name": "Xeon 6",
      "description": "Intel Xeon 6 CPU NUMA node · x86 CPU backend",
      "gpu_count": 4,
      "vram_gb": 1024,
      "multi_node": false,
      "restricted": true
    },
    "alternatives": {
      "multi_node_tp": "/meta-llama/Llama-3.2-1B/strategies/multi_node_tp.json"
    },
    "by_hardware": {
      "xeon6": "/meta-llama/Llama-3.2-1B/hw/xeon6.json"
    }
  },
  "model": {
    "model_id": "meta-llama/Llama-3.2-1B",
    "min_vllm_version": "0.6.2",
    "architecture": "dense",
    "parameter_count": "1.23B",
    "active_parameters": "1.23B",
    "context_length": 131072,
    "base_args": [],
    "base_env": {},
    "install": {
      "pip": {
        "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto"
      },
      "docker": {
        "command": "docker pull vllm/vllm-openai:latest"
      }
    }
  },
  "features": {},
  "opt_in_features": [],
  "variants": {
    "default": {
      "precision": "bf16",
      "vram_minimum_gb": 3,
      "description": "Full precision BF16",
      "supported_hardware": [
        "xeon6"
      ]
    }
  },
  "hardware_overrides": {},
  "guide": "## Overview\n\nLlama 3.2 1B is Meta's compact pretrained dense language model with 128K\ncontext. This recipe includes an Intel Xeon 6 CPU configuration validated\nwith BF16 serving.\n\n## Prerequisites\n\n- Hardware: 1x Xeon 6 NUMA node\n- vLLM >= 0.6.2\n\n### pip (Intel Xeon 6 CPUs)\n\nFor Intel and AMD x86 CPUs, follow the\n[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)\ninstallation instructions.\n\n### Docker (Intel Xeon 6 CPUs)\n\n```bash\ndocker pull vllm/vllm-openai-cpu:latest-x86_64\n```\n\n## Runtime and Platform Tuning\n\nThe following validation settings are not model requirements and are intentionally\nnot portable CPU recipe defaults:\n\n- `--max-num-batched-tokens` and `--max-num-seqs`: scheduler batch/concurrency\n  tuning for workload shape, latency targets, and platform capacity.\n- `--gpu-memory-utilization`: platform memory-budget tuning.\n- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.\n- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: runtime operational timeout tuning.\n\nHardware-specific overrides may still use these settings when they are part of a\nvalidated platform configuration.\n\n## Intel Xeon 6\n\nChoose the tensor parallel size based on the system topology and deployment\nrequirements. Add `--tensor-parallel-size <N>` when needed; the recipe does\nnot prescribe a Xeon 6 TP value or hard-code NUMA node IDs.\n\n```bash\nvllm serve meta-llama/Llama-3.2-1B\n```\n\nDocker:\n\n```bash\ndocker run \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-openai-cpu:latest-x86_64 meta-llama/Llama-3.2-1B \\\n  --tensor-parallel-size 1\n```\n\n## Client Usage\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"unused\")\nresponse = client.completions.create(\n    model=\"meta-llama/Llama-3.2-1B\",\n    prompt=\"Explain tensor parallelism briefly.\",\n    max_tokens=128,\n)\nprint(response.choices[0].text)\n```\n\n## References\n\n- [Model card](https://huggingface.co/meta-llama/Llama-3.2-1B)\n- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)\n",
  "source": {
    "source": "vllm",
    "repo": "vllm-project/vllm",
    "ref": "main",
    "sha": "389ffebe555705c85337025d5df108d0384d365e",
    "path": "recipes/models/meta-llama/Llama-3.2-1B.yaml",
    "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/meta-llama/Llama-3.2-1B.yaml",
    "overlays": []
  }
}