# baidu/ERNIE-4.5-21B-A3B-PT — vLLM serving recipe

> Baidu ERNIE 4.5 MoE text models (21B-A3B, 300B-A47B) with BF16 and FP8 support plus ERNIE-MTP speculative decoding

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16)

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/baidu/ERNIE-4.5-21B-A3B-PT
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/baidu/ERNIE-4.5-21B-A3B-PT.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `300b`
- `strategy`: `single_node_tp`, `multi_node_tp`, `multi_node_tep`, `multi_node_dep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `spec_decoding` (default: none)

## Install

Requires vLLM >= 0.10.1.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve baidu/ERNIE-4.5-21B-A3B-PT \
  --tensor-parallel-size 1
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest baidu/ERNIE-4.5-21B-A3B-PT \
  --tensor-parallel-size 1
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

ERNIE 4.5 is Baidu's MoE language model family. This recipe covers the text-only
variants:

- `baidu/ERNIE-4.5-21B-A3B-PT` — 21B total / 3B active (fits on 1x80GB)
- `baidu/ERNIE-4.5-300B-A47B-PT` — 300B total / 47B active (8x80GB FP8 or 16x80GB BF16)

Both support ERNIE-MTP speculative decoding via `--speculative-config`.

### Prerequisites

- transformers >= 4.54.0
- vLLM >= 0.10.1
- Hardware depends on variant (see above)

#### Install vLLM

```bash
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
```

### Launch commands

21B on 1x80GB GPU:

```bash
vllm serve baidu/ERNIE-4.5-21B-A3B-PT
```

300B on 8x80GB with vLLM FP8 online quantization:

```bash
vllm serve baidu/ERNIE-4.5-300B-A47B-PT \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.95 \
  --quantization fp8
```

300B on 16x80GB native BF16 (multi-node via Ray):

```bash
vllm serve baidu/ERNIE-4.5-300B-A47B-PT --tensor-parallel-size 16
```

ERNIE-MTP speculative decoding (21B example):

```bash
vllm serve baidu/ERNIE-4.5-21B-A3B-PT \
  --speculative-config '{"method":"ernie_mtp","model":"baidu/ERNIE-4.5-21B-A3B-PT","num_speculative_tokens":1}'
```

### Client Usage

Standard OpenAI-compatible API; model ID is the HF repo.

### Benchmarking

```bash
vllm bench serve \
  --model baidu/ERNIE-4.5-21B-A3B-PT \
  --dataset-name random \
  --random-input-len 8000 --random-output-len 1000 \
  --request-rate 10 --num-prompts 16 --ignore-eos
```

Test configurations: prompt-heavy (8k/1k), decode-heavy (1k/8k), balanced (1k/1k).
Vary `--num-prompts` across 1, 16, 32, 64, 128, 256, 512.

### References

- [ERNIE-4.5-21B-A3B-PT](https://huggingface.co/baidu/ERNIE-4.5-21B-A3B-PT)
- [ERNIE-4.5-300B-A47B-PT](https://huggingface.co/baidu/ERNIE-4.5-300B-A47B-PT)
- [vLLM multi-node deployment](https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html)