{ "hf_id": "baidu/ERNIE-4.5-21B-A3B-PT", "meta": { "title": "ERNIE-4.5", "slug": "ernie-4.5", "provider": "Ernie (Baidu)", "description": "Baidu ERNIE 4.5 MoE text models (21B-A3B, 300B-A47B) with BF16 and FP8 support plus ERNIE-MTP speculative decoding", "date_added": "2025-09-03", "date_updated": "2026-04-17", "tasks": [ "text" ], "related_recipes": [ "baidu/ERNIE-4.5-VL-28B-A3B-PT" ] }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve baidu/ERNIE-4.5-21B-A3B-PT \\\n --tensor-parallel-size 1", "argv": [ "vllm", "serve", "baidu/ERNIE-4.5-21B-A3B-PT", "--tensor-parallel-size", "1" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest baidu/ERNIE-4.5-21B-A3B-PT \\\n --tensor-parallel-size 1", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "baidu/ERNIE-4.5-21B-A3B-PT", "--tensor-parallel-size", "1" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy โ€” works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e ยท 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "multi_node_tp": "/baidu/ERNIE-4.5-21B-A3B-PT/strategies/multi_node_tp.json", "multi_node_tep": "/baidu/ERNIE-4.5-21B-A3B-PT/strategies/multi_node_tep.json", "multi_node_dep": "/baidu/ERNIE-4.5-21B-A3B-PT/strategies/multi_node_dep.json", "kv_store_centralized_mooncake": "/baidu/ERNIE-4.5-21B-A3B-PT/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/baidu/ERNIE-4.5-21B-A3B-PT/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/baidu/ERNIE-4.5-21B-A3B-PT/hw/h200.json", "h100": "/baidu/ERNIE-4.5-21B-A3B-PT/hw/h100.json", "b200": "/baidu/ERNIE-4.5-21B-A3B-PT/hw/b200.json", "gb200": "/baidu/ERNIE-4.5-21B-A3B-PT/hw/gb200.json", "b300": "/baidu/ERNIE-4.5-21B-A3B-PT/hw/b300.json", "gb300": "/baidu/ERNIE-4.5-21B-A3B-PT/hw/gb300.json", "mi300x": "/baidu/ERNIE-4.5-21B-A3B-PT/hw/mi300x.json", "mi325x": "/baidu/ERNIE-4.5-21B-A3B-PT/hw/mi325x.json", "mi350x": "/baidu/ERNIE-4.5-21B-A3B-PT/hw/mi350x.json", "mi355x": "/baidu/ERNIE-4.5-21B-A3B-PT/hw/mi355x.json" } }, "model": { "model_id": "baidu/ERNIE-4.5-21B-A3B-PT", "min_vllm_version": "0.10.1", "architecture": "moe", "parameter_count": "21B", "active_parameters": "3B", "context_length": 131072, "base_args": [], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "spec_decoding": { "description": "ERNIE-MTP (multi-token prediction) speculative decoding", "args": [ "--speculative-config", "{\"method\":\"ernie_mtp\",\"model\":\"baidu/ERNIE-4.5-21B-A3B-PT\",\"num_speculative_tokens\":1}" ] } }, "opt_in_features": [ "spec_decoding" ], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 106, "description": "BF16 weights; fits on 1x80GB GPU (21B variant)" }, "300b": { "model_id": "baidu/ERNIE-4.5-300B-A47B-PT", "precision": "bf16", "vram_minimum_gb": 640, "description": "300B total / 47B active; 8x80GB with FP8 online, 16x80GB for BF16", "extra_args": [ "--tensor-parallel-size", "8", "--gpu-memory-utilization", "0.95", "--quantization", "fp8" ], "json": "/baidu/ERNIE-4.5-300B-A47B-PT.json" } }, "kv_offload_support": { "offloading_cpu": "verified", "offloading_fs": "verified" }, "hardware_overrides": { "amd": { "extra_args": [ "--gpu-memory-utilization", "0.9" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1", "SAFETENSORS_FAST_GPU": "1" } } }, "guide": "## Overview\n\nERNIE 4.5 is Baidu's MoE language model family. This recipe covers the text-only\nvariants:\n\n- `baidu/ERNIE-4.5-21B-A3B-PT` โ€” 21B total / 3B active (fits on 1x80GB)\n- `baidu/ERNIE-4.5-300B-A47B-PT` โ€” 300B total / 47B active (8x80GB FP8 or 16x80GB BF16)\n\nBoth support ERNIE-MTP speculative decoding via `--speculative-config`.\n\n## Prerequisites\n\n- transformers >= 4.54.0\n- vLLM >= 0.10.1\n- Hardware depends on variant (see above)\n\n### Install vLLM\n\n```bash\nuv venv --python 3.12 --seed\nsource .venv/bin/activate\nuv pip install vllm --torch-backend=auto\n```\n\n## Launch commands\n\n21B on 1x80GB GPU:\n\n```bash\nvllm serve baidu/ERNIE-4.5-21B-A3B-PT\n```\n\n300B on 8x80GB with vLLM FP8 online quantization:\n\n```bash\nvllm serve baidu/ERNIE-4.5-300B-A47B-PT \\\n --tensor-parallel-size 8 \\\n --gpu-memory-utilization 0.95 \\\n --quantization fp8\n```\n\n300B on 16x80GB native BF16 (multi-node via Ray):\n\n```bash\nvllm serve baidu/ERNIE-4.5-300B-A47B-PT --tensor-parallel-size 16\n```\n\nERNIE-MTP speculative decoding (21B example):\n\n```bash\nvllm serve baidu/ERNIE-4.5-21B-A3B-PT \\\n --speculative-config '{\"method\":\"ernie_mtp\",\"model\":\"baidu/ERNIE-4.5-21B-A3B-PT\",\"num_speculative_tokens\":1}'\n```\n\n## Client Usage\n\nStandard OpenAI-compatible API; model ID is the HF repo.\n\n## Benchmarking\n\n```bash\nvllm bench serve \\\n --model baidu/ERNIE-4.5-21B-A3B-PT \\\n --dataset-name random \\\n --random-input-len 8000 --random-output-len 1000 \\\n --request-rate 10 --num-prompts 16 --ignore-eos\n```\n\nTest configurations: prompt-heavy (8k/1k), decode-heavy (1k/8k), balanced (1k/1k).\nVary `--num-prompts` across 1, 16, 32, 64, 128, 256, 512.\n\n## References\n\n- [ERNIE-4.5-21B-A3B-PT](https://huggingface.co/baidu/ERNIE-4.5-21B-A3B-PT)\n- [ERNIE-4.5-300B-A47B-PT](https://huggingface.co/baidu/ERNIE-4.5-300B-A47B-PT)\n- [vLLM multi-node deployment](https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/baidu/ERNIE-4.5-21B-A3B-PT.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/baidu/ERNIE-4.5-21B-A3B-PT.yaml", "overlays": [] } }