{
  "hf_id": "zai-org/GLM-4.6V",
  "meta": {
    "title": "GLM-4.6V",
    "slug": "glm-4.6v",
    "provider": "GLM (Z-AI)",
    "description": "GLM-4.6 vision-language MoE model — image-text-to-text with 128K context, native FP8 checkpoint, and expert parallelism",
    "date_added": "2025-12-07",
    "date_updated": "2026-04-17",
    "tasks": [
      "multimodal"
    ],
    "performance_headline": "Updated GLM-V series with 128K context length and native FP8",
    "related_recipes": [],
    "hardware": {
      "h100": "verified",
      "h200": "verified",
      "mi300x": "verified",
      "mi325x": "verified",
      "mi355x": "verified"
    }
  },
  "recommended_command": {
    "hardware": "h200",
    "strategy": "single_node_tp",
    "variant": "default",
    "node_count": 1,
    "deploy_type": "single_node",
    "env": {},
    "docker_image": "vllm/vllm-openai:latest",
    "command": "vllm serve zai-org/GLM-4.6V \\\n  --trust-remote-code \\\n  --enable-expert-parallel \\\n  --allowed-local-media-path / \\\n  --mm-processor-cache-type shm \\\n  --tensor-parallel-size 8 \\\n  --tool-call-parser glm45 \\\n  --enable-auto-tool-choice \\\n  --reasoning-parser glm45 \\\n  --mm-encoder-tp-mode data",
    "argv": [
      "vllm",
      "serve",
      "zai-org/GLM-4.6V",
      "--trust-remote-code",
      "--enable-expert-parallel",
      "--allowed-local-media-path",
      "/",
      "--mm-processor-cache-type",
      "shm",
      "--tensor-parallel-size",
      "8",
      "--tool-call-parser",
      "glm45",
      "--enable-auto-tool-choice",
      "--reasoning-parser",
      "glm45",
      "--mm-encoder-tp-mode",
      "data"
    ],
    "docker_command": "docker run --gpus all \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-openai:latest zai-org/GLM-4.6V \\\n  --trust-remote-code \\\n  --enable-expert-parallel \\\n  --allowed-local-media-path / \\\n  --mm-processor-cache-type shm \\\n  --tensor-parallel-size 8 \\\n  --tool-call-parser glm45 \\\n  --enable-auto-tool-choice \\\n  --reasoning-parser glm45 \\\n  --mm-encoder-tp-mode data",
    "docker_argv": [
      "docker",
      "run",
      "--gpus",
      "all",
      "--privileged",
      "--ipc=host",
      "-p",
      "8000:8000",
      "-v",
      "~/.cache/huggingface:/root/.cache/huggingface",
      "vllm/vllm-openai:latest",
      "zai-org/GLM-4.6V",
      "--trust-remote-code",
      "--enable-expert-parallel",
      "--allowed-local-media-path",
      "/",
      "--mm-processor-cache-type",
      "shm",
      "--tensor-parallel-size",
      "8",
      "--tool-call-parser",
      "glm45",
      "--enable-auto-tool-choice",
      "--reasoning-parser",
      "glm45",
      "--mm-encoder-tp-mode",
      "data"
    ],
    "strategy_spec": {
      "name": "single_node_tp",
      "deploy_type": "single_node",
      "display_name": "Tensor Parallel",
      "short_label": "TP",
      "orientation": "latency",
      "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n",
      "hardware_match": {
        "min_gpus": 1,
        "max_gpus": 8,
        "multi_node": false
      },
      "vllm_args": [],
      "parallel_flag": "--tensor-parallel-size"
    },
    "hardware_profile": {
      "brand": "NVIDIA",
      "generation": "hopper",
      "compute_arch": "sm90",
      "display_name": "H200",
      "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node",
      "gpu_count": 8,
      "vram_gb": 1128,
      "multi_node": false
    },
    "alternatives": {
      "single_node_tep": "/zai-org/GLM-4.6V/strategies/single_node_tep.json",
      "single_node_dep": "/zai-org/GLM-4.6V/strategies/single_node_dep.json",
      "multi_node_tp": "/zai-org/GLM-4.6V/strategies/multi_node_tp.json",
      "multi_node_tp_pp": "/zai-org/GLM-4.6V/strategies/multi_node_tp_pp.json",
      "multi_node_tep": "/zai-org/GLM-4.6V/strategies/multi_node_tep.json",
      "multi_node_dep": "/zai-org/GLM-4.6V/strategies/multi_node_dep.json",
      "kv_store_centralized_mooncake": "/zai-org/GLM-4.6V/strategies/kv_store_centralized_mooncake.json",
      "kv_store_distributed_mooncake": "/zai-org/GLM-4.6V/strategies/kv_store_distributed_mooncake.json"
    },
    "by_hardware": {
      "h200": "/zai-org/GLM-4.6V/hw/h200.json",
      "h100": "/zai-org/GLM-4.6V/hw/h100.json",
      "b200": "/zai-org/GLM-4.6V/hw/b200.json",
      "gb200": "/zai-org/GLM-4.6V/hw/gb200.json",
      "b300": "/zai-org/GLM-4.6V/hw/b300.json",
      "gb300": "/zai-org/GLM-4.6V/hw/gb300.json",
      "mi300x": "/zai-org/GLM-4.6V/hw/mi300x.json",
      "mi325x": "/zai-org/GLM-4.6V/hw/mi325x.json",
      "mi350x": "/zai-org/GLM-4.6V/hw/mi350x.json",
      "mi355x": "/zai-org/GLM-4.6V/hw/mi355x.json"
    }
  },
  "model": {
    "model_id": "zai-org/GLM-4.6V",
    "min_vllm_version": "0.12.0",
    "architecture": "moe",
    "parameter_count": "107B",
    "active_parameters": "12B",
    "context_length": 131072,
    "base_args": [
      "--trust-remote-code",
      "--enable-expert-parallel",
      "--allowed-local-media-path",
      "/",
      "--mm-encoder-tp-mode",
      "data",
      "--mm-processor-cache-type",
      "shm"
    ],
    "base_env": {},
    "install": {
      "pip": {
        "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto"
      },
      "docker": {
        "command": "docker pull vllm/vllm-openai:latest"
      },
      "extras": [
        {
          "note": "GLM-V requires Transformers >= 5.0.0 for tokenizer and multimodal config compatibility.",
          "command": "uv pip install \"transformers>=5.0.0\""
        }
      ]
    }
  },
  "features": {
    "tool_calling": {
      "description": "GLM-4.5 tool call parser with automatic tool choice",
      "args": [
        "--tool-call-parser",
        "glm45",
        "--enable-auto-tool-choice"
      ]
    },
    "reasoning": {
      "description": "GLM-4.5 reasoning parser",
      "args": [
        "--reasoning-parser",
        "glm45"
      ]
    },
    "text_only": {
      "description": "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel.",
      "args": [
        "--language-model-only"
      ]
    },
    "encoder_parallel": {
      "description": "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only.",
      "args": [
        "--mm-encoder-tp-mode",
        "data"
      ]
    }
  },
  "opt_in_features": [
    "text_only"
  ],
  "variants": {
    "default": {
      "precision": "bf16",
      "vram_minimum_gb": 257,
      "description": "Full precision BF16 — runs on 4xH100/H200"
    },
    "fp8": {
      "model_id": "zai-org/GLM-4.6V-FP8",
      "precision": "fp8",
      "vram_minimum_gb": 128,
      "description": "Native FP8 checkpoint with minimal accuracy loss",
      "json": "/zai-org/GLM-4.6V-FP8.json"
    }
  },
  "kv_offload_support": {
    "offloading_cpu": "verified",
    "offloading_fs": "verified"
  },
  "hardware_overrides": {
    "amd": {
      "extra_args": [
        "--mm-encoder-tp-mode",
        "data",
        "--allowed-local-media-path",
        "/"
      ],
      "extra_env": {
        "VLLM_ROCM_USE_AITER": "1",
        "SAFETENSORS_FAST_GPU": "1"
      }
    }
  },
  "guide": "## Overview\n\nGLM-4.6V is an updated vision-language MoE model from Z-AI. It supports a 128K\ncontext length (vs 64K for GLM-4.5V). Native FP8 is recommended for cost-efficient\nserving, matching BF16 accuracy to within a small margin.\n\n## Prerequisites\n\n- **vLLM version:** >= 0.12.0\n- **Hardware:** 4x H100/H200 (BF16 or FP8), or 4x MI300X/MI325X/MI355X with ROCm\n- **Python:** 3.10+ for NVIDIA; Python 3.12 with ROCm 7.0 and glibc >= 2.35 for AMD ROCm wheels\n\n### Install vLLM (NVIDIA)\n\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto\n```\n\n### Install vLLM (AMD ROCm)\n\n```bash\nuv venv --python 3.12\nsource .venv/bin/activate\nuv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm\nuv pip install \"transformers>=5.0.0\"\n```\n\n## Launching the Server\n\n### Tensor Parallel + Expert Parallel (FP8 on 4 GPUs)\n\n```bash\nvllm serve zai-org/GLM-4.6V-FP8 \\\n     --tensor-parallel-size 4 \\\n     --tool-call-parser glm45 \\\n     --reasoning-parser glm45 \\\n     --enable-auto-tool-choice \\\n     --enable-expert-parallel \\\n     --allowed-local-media-path / \\\n     --mm-encoder-tp-mode data \\\n     --mm-processor-cache-type shm\n```\n\n### AMD ROCm (FP8 on 4 GPUs)\n\n```bash\nexport SAFETENSORS_FAST_GPU=1\nexport VLLM_ROCM_USE_AITER=1\n\nvllm serve zai-org/GLM-4.6V-FP8 \\\n     --tensor-parallel-size 4 \\\n     --tool-call-parser glm45 \\\n     --reasoning-parser glm45 \\\n     --enable-auto-tool-choice \\\n     --enable-expert-parallel \\\n     --allowed-local-media-path / \\\n     --mm-encoder-tp-mode data \\\n     --mm-processor-cache-type shm\n```\n\n### Tuning Tips\n\n- `--max-model-len=65536` is a common default; you can push to 131072.\n- `--max-num-batched-tokens=32768` for prompt-heavy workloads.\n- `--mm-encoder-tp-mode data` + `--mm-processor-cache-type shm` for efficient vision processing.\n- Use `uv pip install \"transformers>=5.0.0\"` before serving GLM-V checkpoints.\n\n## Client Usage\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\nresp = client.chat.completions.create(\n    model=\"zai-org/GLM-4.6V-FP8\",\n    messages=[{\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image_url\", \"image_url\": {\"url\": \"https://example.com/image.png\"}},\n            {\"type\": \"text\", \"text\": \"Describe the image.\"}\n        ]\n    }],\n    max_tokens=512,\n)\nprint(resp.choices[0].message.content)\n```\n\n## Benchmarking\n\n```bash\nvllm bench serve \\\n  --backend openai-chat \\\n  --endpoint /v1/chat/completions \\\n  --model zai-org/GLM-4.6V-FP8 \\\n  --dataset-name hf \\\n  --dataset-path lmarena-ai/VisionArena-Chat \\\n  --num-prompts 1000 \\\n  --request-rate 20\n```\n\n## Troubleshooting\n\n- **Vision encoder overhead:** Prefer `--mm-encoder-tp-mode data` over TP for the encoder.\n- **Long-context memory:** At 128K context, tune `--max-num-batched-tokens` and\n  `--gpu-memory-utilization` to prevent OOM.\n\n## References\n\n- [Model card](https://huggingface.co/zai-org/GLM-4.6V)\n- [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)\n",
  "source": {
    "source": "vllm",
    "repo": "vllm-project/vllm",
    "ref": "main",
    "sha": "389ffebe555705c85337025d5df108d0384d365e",
    "path": "recipes/models/zai-org/GLM-4.6V.yaml",
    "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/zai-org/GLM-4.6V.yaml",
    "overlays": []
  }
}