{
  "hf_id": "Google/gemma-4-E4B-it",
  "meta": {
    "title": "Gemma 4 E4B IT",
    "slug": "gemma-4-e4b-it",
    "provider": "Google",
    "description": "Google's compact Gemma 4 multimodal model (effective 4B) with native text, image, and audio, plus thinking mode and tool-use protocol.",
    "date_added": "2026-04-02",
    "date_updated": "2026-09-12",
    "tasks": [
      "multimodal",
      "text"
    ],
    "performance_headline": "Effective-4B unified multimodal model with audio, thinking, and function calling — runs on a single 24 GB+ GPU",
    "related_recipes": [
      "google/gemma-4-E2B-it",
      "google/gemma-4-31B-it",
      "google/gemma-4-26B-A4B-it"
    ],
    "hardware": {
      "h100": "verified",
      "mi300x": "verified",
      "mi325x": "verified",
      "mi355x": "verified",
      "trillium": "verified",
      "ironwood": "verified",
      "xeon6": "verified"
    }
  },
  "recommended_command": {
    "hardware": "h100",
    "strategy": "single_node_tp",
    "variant": "default",
    "node_count": 1,
    "deploy_type": "single_node",
    "env": {},
    "docker_image": "vllm/vllm-openai:latest",
    "command": "vllm serve google/gemma-4-E4B-it \\\n  --tensor-parallel-size 1 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser gemma4 \\\n  --chat-template examples/tool_chat_template_gemma4.jinja \\\n  --reasoning-parser gemma4",
    "argv": [
      "vllm",
      "serve",
      "google/gemma-4-E4B-it",
      "--tensor-parallel-size",
      "1",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "gemma4",
      "--chat-template",
      "examples/tool_chat_template_gemma4.jinja",
      "--reasoning-parser",
      "gemma4"
    ],
    "docker_command": "docker run --gpus all \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-openai:latest google/gemma-4-E4B-it \\\n  --tensor-parallel-size 1 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser gemma4 \\\n  --chat-template examples/tool_chat_template_gemma4.jinja \\\n  --reasoning-parser gemma4",
    "docker_argv": [
      "docker",
      "run",
      "--gpus",
      "all",
      "--privileged",
      "--ipc=host",
      "-p",
      "8000:8000",
      "-v",
      "~/.cache/huggingface:/root/.cache/huggingface",
      "vllm/vllm-openai:latest",
      "google/gemma-4-E4B-it",
      "--tensor-parallel-size",
      "1",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "gemma4",
      "--chat-template",
      "examples/tool_chat_template_gemma4.jinja",
      "--reasoning-parser",
      "gemma4"
    ],
    "strategy_spec": {
      "name": "single_node_tp",
      "deploy_type": "single_node",
      "display_name": "Tensor Parallel",
      "short_label": "TP",
      "orientation": "latency",
      "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n",
      "hardware_match": {
        "min_gpus": 1,
        "max_gpus": 8,
        "multi_node": false
      },
      "vllm_args": [],
      "parallel_flag": "--tensor-parallel-size"
    },
    "hardware_profile": {
      "brand": "NVIDIA",
      "generation": "hopper",
      "compute_arch": "sm90",
      "display_name": "H100",
      "description": "NVIDIA H100 80GB NVLink (8-GPU node)",
      "gpu_count": 8,
      "vram_gb": 640,
      "multi_node": false
    },
    "alternatives": {
      "multi_node_tp": "/Google/gemma-4-E4B-it/strategies/multi_node_tp.json",
      "kv_store_centralized_mooncake": "/Google/gemma-4-E4B-it/strategies/kv_store_centralized_mooncake.json",
      "kv_store_distributed_mooncake": "/Google/gemma-4-E4B-it/strategies/kv_store_distributed_mooncake.json"
    },
    "by_hardware": {
      "h100": "/Google/gemma-4-E4B-it/hw/h100.json",
      "mi300x": "/Google/gemma-4-E4B-it/hw/mi300x.json",
      "mi325x": "/Google/gemma-4-E4B-it/hw/mi325x.json",
      "mi355x": "/Google/gemma-4-E4B-it/hw/mi355x.json",
      "trillium": "/Google/gemma-4-E4B-it/hw/trillium.json",
      "ironwood": "/Google/gemma-4-E4B-it/hw/ironwood.json",
      "xeon6": "/Google/gemma-4-E4B-it/hw/xeon6.json"
    }
  },
  "model": {
    "model_id": "google/gemma-4-E4B-it",
    "min_vllm_version": "0.19.1",
    "architecture": "dense",
    "parameter_count": "8B",
    "active_parameters": "8B",
    "context_length": 131072,
    "base_args": [],
    "base_env": {},
    "install": {
      "pip": {
        "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto"
      },
      "docker": {
        "command": "docker pull vllm/vllm-openai:latest"
      },
      "extras": [
        {
          "note": "Audio extras — needed to serve the audio modality",
          "command": "uv pip install \"vllm[audio]\"",
          "optional": true
        }
      ]
    }
  },
  "features": {
    "tool_calling": {
      "description": "Enable automatic tool choice with Gemma 4 parser and chat template",
      "args": [
        "--enable-auto-tool-choice",
        "--tool-call-parser",
        "gemma4",
        "--chat-template",
        "examples/tool_chat_template_gemma4.jinja"
      ]
    },
    "reasoning": {
      "description": "Enable structured thinking/reasoning output",
      "args": [
        "--reasoning-parser",
        "gemma4"
      ]
    },
    "text_only": {
      "description": "Skip loading the vision and audio encoders for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel.",
      "args": [
        "--language-model-only"
      ]
    },
    "spec_decoding": {
      "description": "MTP speculative decoding with centroids masking for accelerated inference",
      "args": [
        "--speculative-config",
        "{\"model\":\"google/gemma-4-E4B-it-assistant\",\"num_speculative_tokens\":4}"
      ]
    }
  },
  "opt_in_features": [
    "text_only",
    "spec_decoding"
  ],
  "variants": {
    "default": {
      "precision": "bf16",
      "vram_minimum_gb": 20,
      "description": "Full BF16 — single 24 GB+ NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355Xi or 2x Xeon6/Xeon5 NUMA nodes",
      "supported_hardware": [
        "h100",
        "mi300x",
        "mi325x",
        "mi355x",
        "trillium",
        "ironwood",
        "xeon6"
      ]
    },
    "fp8": {
      "model_id": "RedHatAI/gemma-4-E4B-it-FP8-dynamic",
      "precision": "fp8",
      "vram_minimum_gb": 10,
      "description": "FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision/audio encoders stay BF16) — Hopper or Blackwell",
      "supported_hardware": [
        "h100"
      ],
      "extra_args": [
        "--kv-cache-dtype",
        "fp8"
      ],
      "json": "/RedHatAI/gemma-4-E4B-it-FP8-dynamic.json"
    },
    "w4a16": {
      "model_id": "google/gemma-4-E4B-it-qat-w4a16-ct",
      "precision": "int4",
      "vram_minimum_gb": 10,
      "description": "W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU",
      "supported_hardware": [
        "h100",
        "mi300x",
        "mi325x",
        "mi355x"
      ],
      "json": "/google/gemma-4-E4B-it-qat-w4a16-ct.json"
    },
    "mobile": {
      "model_id": "google/gemma-4-E4B-it-qat-mobile-ct",
      "precision": "int2/4/8",
      "vram_minimum_gb": 5,
      "description": "Mobile QAT — mixed int2/4/8 compressed-tensors (int2 embeddings + lm_head + audio, int4 LLM attention + MLP, int8 vision + gates). 4.1× weight compression vs BF16, up to 1.5× faster decode at low batch",
      "supported_hardware": [
        "h100",
        "mi300x",
        "mi325x",
        "mi355x"
      ],
      "json": "/google/gemma-4-E4B-it-qat-mobile-ct.json"
    }
  },
  "guide": "## Overview\n\n[Gemma 4 E4B](https://huggingface.co/google/gemma-4-E4B-it) is Google's effective-4B unified multimodal model — text + images + audio in a single model, with structured thinking/reasoning, function calling, and dynamic vision resolution. It fits on a single 24 GB+ GPU.\n\nIntel Xeon 6 CPU support remains verified. No additional benchmark-specific CPU\ntuning is prescribed for general deployment.\n\n### Key Features\n- **Multimodal**: Text + images + audio natively (video via custom frame-extraction pipeline).\n- **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions.\n- **Thinking Mode**: Structured reasoning via `<|channel>thought\\n...<channel|>` delimiters.\n- **Function Calling**: Custom tool-call protocol with dedicated special tokens.\n- **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens).\n\nTPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).\n\n## Prerequisites\n\n- Hardware: Intel Xeon 6/Xeon 5 CPUs\n- vLLM >= 0.19.1\n\n### pip (NVIDIA CUDA)\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --pre \\\n  --extra-index-url https://wheels.vllm.ai/nightly/cu129 \\\n  --extra-index-url https://download.pytorch.org/whl/cu129 \\\n  --index-strategy unsafe-best-match\n```\n\n### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X)\nRequires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+).\n```bash\nuv venv --python 3.12\nsource .venv/bin/activate\nuv pip install vllm --pre \\\n  --extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade\n```\n\n### pip (Intel Xeon 6 CPUs)\nFor Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.\n\n### Docker\n```bash\ndocker pull vllm/vllm-openai:gemma4-0505-cu129  # NVIDIA Hopper (H100/H200, CUDA 12.9)\ndocker pull vllm/vllm-openai:gemma4-0505-cu130  # NVIDIA Blackwell (B200/B300, CUDA 13.0)\ndocker pull vllm/vllm-openai-rocm:latest   # AMD\ndocker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6\n```\n\n## Deployment Configurations\n\n### Quick Start (Single GPU)\n```bash\nvllm serve google/gemma-4-E4B-it \\\n  --max-model-len 32768\n```\n\n### With Audio Support\n```bash\nvllm serve google/gemma-4-E4B-it \\\n  --max-model-len 8192 \\\n  --limit-mm-per-prompt '{\"image\": 4, \"audio\": 1}'\n```\n\n### Full-Featured Server Launch\nEnables text, image, audio, thinking, and tool calling:\n```bash\nvllm serve google/gemma-4-E4B-it \\\n  --max-model-len 16384 \\\n  --gpu-memory-utilization 0.90 \\\n  --enable-auto-tool-choice \\\n  --reasoning-parser gemma4 \\\n  --tool-call-parser gemma4 \\\n  --chat-template examples/tool_chat_template_gemma4.jinja \\\n  --limit-mm-per-prompt '{\"image\": 4, \"audio\": 1}' \\\n  --async-scheduling \\\n  --host 0.0.0.0 \\\n  --port 8000\n```\n\n### Docker (NVIDIA)\n```bash\ndocker run -itd --name gemma4-e4b \\\n  --ipc=host --network host --shm-size 16G --gpus all \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-openai:gemma4-0505-cu129 \\\n    --model google/gemma-4-E4B-it \\\n    --max-model-len 32768 \\\n    --host 0.0.0.0 --port 8000\n```\nSwap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300).\n\n### Docker (AMD MI300X/MI325X/MI350X/MI355X)\n```bash\ndocker run -itd --name gemma4-rocm \\\n  --ipc=host --network=host --privileged \\\n  --cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \\\n  --group-add=video --cap-add=SYS_PTRACE \\\n  --security-opt=seccomp=unconfined --shm-size 16G \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-openai-rocm:latest \\\n    --model google/gemma-4-E4B-it \\\n    --host 0.0.0.0 --port 8000\n```\n\n### Docker (Cloud TPU — Trillium / Ironwood)\nTPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:\n```bash\ndocker run -itd --name gemma4-tpu \\\n  --privileged --network host --shm-size 16G \\\n  -v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \\\n  vllm/vllm-tpu:latest \\\n    --model google/gemma-4-E4B-it \\\n    --max-model-len 16384 \\\n    --disable_chunked_mm_input \\\n    --host 0.0.0.0 --port 8000\n```\n\n### Intel Xeon 6 Deployment via Docker\n\nLaunch the x86 CPU vLLM Docker container for `google/gemma-4-E4B-it`:\n\n```bash\ndocker run \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-openai-cpu:latest-x86_64 google/gemma-4-E4B-it \\\n  --tensor-parallel-size 1 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser gemma4 \\\n  --chat-template examples/tool_chat_template_gemma4.jinja \\\n  --reasoning-parser gemma4\n```\n\nFor additional Intel Xeon 6 deployment details, see the Intel Software Catalog entries for [Gemma 4 E4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-e4b-it).\n\nThe following settings are intentionally not prescribed as portable CPU model\ndefaults because they depend on the workload, hardware, or runtime environment:\n\n- `--max-num-batched-tokens`: scheduler/throughput tuning.\n- `--max-num-seqs`: concurrency and scheduler-capacity tuning.\n- `--gpu-memory-utilization`: platform memory-budget tuning.\n- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.\n- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.\n\nThese settings may still appear in validated hardware-specific overrides. Tune\nthem at deployment time based on platform resources, workload shape, and\nlatency/throughput goals.\n\n## Client Usage\n\n### Audio Transcription\n```python\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"EMPTY\")\nresponse = client.chat.completions.create(\n    model=\"google/gemma-4-E4B-it\",\n    messages=[{\"role\": \"user\", \"content\": [\n        {\"type\": \"audio_url\", \"audio_url\": {\"url\": \"https://example.com/audio.wav\"}},\n        {\"type\": \"text\", \"text\": \"Transcribe this audio.\"},\n    ]}],\n    max_tokens=512,\n)\nprint(response.choices[0].message.content)\n```\n\n### Image Understanding\n```python\nresponse = client.chat.completions.create(\n    model=\"google/gemma-4-E4B-it\",\n    messages=[{\"role\": \"user\", \"content\": [\n        {\"type\": \"image_url\", \"image_url\": {\"url\": \"https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg\"}},\n        {\"type\": \"text\", \"text\": \"Describe this image in detail.\"},\n    ]}],\n    max_tokens=1024,\n)\n```\n\n### Thinking Mode\n```bash\nvllm serve google/gemma-4-E4B-it \\\n  --max-model-len 16384 \\\n  --reasoning-parser gemma4 \\\n  --tool-call-parser gemma4 \\\n  --enable-auto-tool-choice \\\n  --chat-template examples/tool_chat_template_gemma4.jinja\n```\nEnable per-request via `extra_body={\"chat_template_kwargs\": {\"enable_thinking\": True}}`.\n\n## Configuration Tips\n\n- Set `--max-model-len` to match your workload (max 131072).\n- Image-only workloads: `--limit-mm-per-prompt.audio 0`.\n- Text-only workloads: `--limit-mm-per-prompt '{\"image\": 0, \"audio\": 0}'` to skip MM profiling.\n- `--async-scheduling` improves throughput.\n- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.\n\n## Quantized Variant\n\n[`RedHatAI/gemma-4-E4B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-E4B-it-FP8-dynamic) is a pre-quantized FP8 (E4M3) checkpoint — linear weights with dynamic per-token activation quantization, vision/audio encoders kept in BF16. Runs on Hopper and Blackwell. Pick the **fp8** variant above, or pass the repo id directly to `vllm serve`.\n\n## Speculative Decoding (MTP)\n\nEnable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-E4B-it-assistant). Recommended `num_speculative_tokens`: 4 for this model. The E4B assistant uses centroids masking for efficient sparse logit computation. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks.\n\n> **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature.\n\n## References\n\n- [Model card](https://huggingface.co/google/gemma-4-E4B-it)\n- [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-E4B-it-FP8-dynamic)\n- [Gemma docs](https://ai.google.dev/gemma/docs)\n- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)\n- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)\n- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)\n- [Xeon 6 recipe: Gemma 4 E4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-e4b-it)\n",
  "source": {
    "source": "vllm",
    "repo": "vllm-project/vllm",
    "ref": "main",
    "sha": "389ffebe555705c85337025d5df108d0384d365e",
    "path": "recipes/models/Google/gemma-4-E4B-it.yaml",
    "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/Google/gemma-4-E4B-it.yaml",
    "overlays": []
  }
}