# inclusionAI/Ring-1T-FP8 — vLLM serving recipe

> Ring-1T (BailingMoeV2) FP8 model (~1T total params) for 8xH200 or 8xMI300X deployment

Configuration: hardware `h200` (8× H200) · strategy `multi_node_tp` (Multi-Node Tensor Parallel) · 2 nodes · variant `default` (FP8)

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/inclusionAI/Ring-1T-FP8
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/inclusionAI/Ring-1T-FP8.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `strategy`: `single_node_tp`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_tep`, `multi_node_dep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)

## Install

Requires vLLM >= 0.11.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

Run one command per node: the head (node rank 0) plus 1 worker. Each node's command differs — don't replay one on every node.

### Head (node 0)

```bash
export GLOO_SOCKET_IFNAME=$IFACE_NAME
export NCCL_SOCKET_IFNAME=$IFACE_NAME
vllm serve inclusionAI/Ring-1T-FP8 \
  --trust-remote-code \
  --max-num-seqs 32 \
  --kv-cache-dtype fp8 \
  --served-model-name Ring-1T-FP8 \
  --gpu-memory-utilization 0.97 \
  --compilation-config '{"use_inductor": false}' \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_IP
```

### Node 1

```bash
export GLOO_SOCKET_IFNAME=$IFACE_NAME
export NCCL_SOCKET_IFNAME=$IFACE_NAME
vllm serve inclusionAI/Ring-1T-FP8 \
  --trust-remote-code \
  --max-num-seqs 32 \
  --kv-cache-dtype fp8 \
  --served-model-name Ring-1T-FP8 \
  --gpu-memory-utilization 0.97 \
  --compilation-config '{"use_inductor": false}' \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 1 \
  --master-addr $HEAD_IP \
  --headless
```

Docker equivalent:

```bash
# Head
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e GLOO_SOCKET_IFNAME=$IFACE_NAME \
  -e NCCL_SOCKET_IFNAME=$IFACE_NAME \
  vllm/vllm-openai:latest inclusionAI/Ring-1T-FP8 \
  --trust-remote-code \
  --max-num-seqs 32 \
  --kv-cache-dtype fp8 \
  --served-model-name Ring-1T-FP8 \
  --gpu-memory-utilization 0.97 \
  --compilation-config '{"use_inductor": false}' \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_IP
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Ring-1T-FP8 is inclusionAI's BailingMoeV2 FP8 model (~1T total parameters). This recipe
covers pure tensor-parallel deployment across 8 GPUs on NVIDIA H200 or AMD MI300X+.

### Prerequisites

- Hardware: 8x H200 or 8x MI300X/MI325X/MI355X
- vLLM >= 0.11.0

#### Install vLLM (CUDA)

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### Install vLLM (AMD ROCm)

```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700
```

### Launch commands

8x H200 (FP8 KV cache):

```bash
vllm serve inclusionAI/Ring-1T-FP8 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.97 \
  --max-num-seqs 32 \
  --kv-cache-dtype fp8 \
  --compilation-config '{"use_inductor": false}' \
  --served-model-name Ring-1T-FP8
```

8x MI300X/MI325X/MI355X:

```bash
export VLLM_ROCM_USE_AITER=1
vllm serve inclusionAI/Ring-1T-FP8 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.9 \
  --max-num-seqs 32 \
  --kv-cache-dtype fp8 \
  --served-model-name Ring-1T-FP8
```

Tuning flags:
- `--max-model-len=65536` works well for most scenarios.
- `--max-num-batched-tokens=32768` for prompt-heavy; 16384/8192 for lower latency.
- Reduce `--gpu-memory-utilization` below 0.97 if you hit OOM.

### Client Usage

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Ring-1T-FP8",
    "messages": [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]
  }'
```

### References

- [Ring-1T-FP8 on Hugging Face](https://huggingface.co/inclusionAI/Ring-1T-FP8)