{ "hf_id": "inclusionAI/Ring-1T-FP8", "meta": { "title": "Ring-1T-FP8", "slug": "ring-1t-fp8", "provider": "inclusionAI", "description": "Ring-1T (BailingMoeV2) FP8 model (~1T total params) for 8xH200 or 8xMI300X deployment", "date_added": "2025-10-15", "date_updated": "2026-04-17", "tasks": [ "text" ], "related_recipes": [], "hardware": { "h200": "verified", "mi300x": "verified", "mi325x": "verified", "mi355x": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "multi_node_tp", "variant": "default", "node_count": 2, "deploy_type": "multi_node", "env": { "GLOO_SOCKET_IFNAME": "$IFACE_NAME", "NCCL_SOCKET_IFNAME": "$IFACE_NAME" }, "docker_image": "vllm/vllm-openai:latest", "head_command": "vllm serve inclusionAI/Ring-1T-FP8 \\\n --trust-remote-code \\\n --max-num-seqs 32 \\\n --kv-cache-dtype fp8 \\\n --served-model-name Ring-1T-FP8 \\\n --gpu-memory-utilization 0.97 \\\n --compilation-config '{\"use_inductor\": false}' \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 0 \\\n --master-addr $HEAD_IP", "worker_command": "vllm serve inclusionAI/Ring-1T-FP8 \\\n --trust-remote-code \\\n --max-num-seqs 32 \\\n --kv-cache-dtype fp8 \\\n --served-model-name Ring-1T-FP8 \\\n --gpu-memory-utilization 0.97 \\\n --compilation-config '{\"use_inductor\": false}' \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless", "worker_commands": [ "vllm serve inclusionAI/Ring-1T-FP8 \\\n --trust-remote-code \\\n --max-num-seqs 32 \\\n --kv-cache-dtype fp8 \\\n --served-model-name Ring-1T-FP8 \\\n --gpu-memory-utilization 0.97 \\\n --compilation-config '{\"use_inductor\": false}' \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless" ], "head_argv": [ "vllm", "serve", "inclusionAI/Ring-1T-FP8", "--trust-remote-code", "--max-num-seqs", "32", "--kv-cache-dtype", "fp8", "--served-model-name", "Ring-1T-FP8", "--gpu-memory-utilization", "0.97", "--compilation-config", "{\"use_inductor\": false}", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "0", "--master-addr", "$HEAD_IP" ], "worker_argv": [ "vllm", "serve", "inclusionAI/Ring-1T-FP8", "--trust-remote-code", "--max-num-seqs", "32", "--kv-cache-dtype", "fp8", "--served-model-name", "Ring-1T-FP8", "--gpu-memory-utilization", "0.97", "--compilation-config", "{\"use_inductor\": false}", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless" ], "worker_argvs": [ [ "vllm", "serve", "inclusionAI/Ring-1T-FP8", "--trust-remote-code", "--max-num-seqs", "32", "--kv-cache-dtype", "fp8", "--served-model-name", "Ring-1T-FP8", "--gpu-memory-utilization", "0.97", "--compilation-config", "{\"use_inductor\": false}", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless" ] ], "head_docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n vllm/vllm-openai:latest inclusionAI/Ring-1T-FP8 \\\n --trust-remote-code \\\n --max-num-seqs 32 \\\n --kv-cache-dtype fp8 \\\n --served-model-name Ring-1T-FP8 \\\n --gpu-memory-utilization 0.97 \\\n --compilation-config '{\"use_inductor\": false}' \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 0 \\\n --master-addr $HEAD_IP", "worker_docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n vllm/vllm-openai:latest inclusionAI/Ring-1T-FP8 \\\n --trust-remote-code \\\n --max-num-seqs 32 \\\n --kv-cache-dtype fp8 \\\n --served-model-name Ring-1T-FP8 \\\n --gpu-memory-utilization 0.97 \\\n --compilation-config '{\"use_inductor\": false}' \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless", "worker_docker_commands": [ "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n vllm/vllm-openai:latest inclusionAI/Ring-1T-FP8 \\\n --trust-remote-code \\\n --max-num-seqs 32 \\\n --kv-cache-dtype fp8 \\\n --served-model-name Ring-1T-FP8 \\\n --gpu-memory-utilization 0.97 \\\n --compilation-config '{\"use_inductor\": false}' \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless" ], "head_docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "GLOO_SOCKET_IFNAME=$IFACE_NAME", "-e", "NCCL_SOCKET_IFNAME=$IFACE_NAME", "vllm/vllm-openai:latest", "inclusionAI/Ring-1T-FP8", "--trust-remote-code", "--max-num-seqs", "32", "--kv-cache-dtype", "fp8", "--served-model-name", "Ring-1T-FP8", "--gpu-memory-utilization", "0.97", "--compilation-config", "{\"use_inductor\": false}", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "0", "--master-addr", "$HEAD_IP" ], "worker_docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "GLOO_SOCKET_IFNAME=$IFACE_NAME", "-e", "NCCL_SOCKET_IFNAME=$IFACE_NAME", "vllm/vllm-openai:latest", "inclusionAI/Ring-1T-FP8", "--trust-remote-code", "--max-num-seqs", "32", "--kv-cache-dtype", "fp8", "--served-model-name", "Ring-1T-FP8", "--gpu-memory-utilization", "0.97", "--compilation-config", "{\"use_inductor\": false}", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless" ], "worker_docker_argvs": [ [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "GLOO_SOCKET_IFNAME=$IFACE_NAME", "-e", "NCCL_SOCKET_IFNAME=$IFACE_NAME", "vllm/vllm-openai:latest", "inclusionAI/Ring-1T-FP8", "--trust-remote-code", "--max-num-seqs", "32", "--kv-cache-dtype", "fp8", "--served-model-name", "Ring-1T-FP8", "--gpu-memory-utilization", "0.97", "--compilation-config", "{\"use_inductor\": false}", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless" ] ], "strategy_spec": { "name": "multi_node_tp", "deploy_type": "multi_node", "display_name": "Multi-Node Tensor Parallel", "short_label": "Multi-node TP", "orientation": "latency", "description": "Multi-node TP. Splits the model across GPUs spanning multiple nodes. TP is set to total GPU count across all nodes. Head node serves requests, worker nodes run with --headless. Requires InfiniBand interconnect.\n", "hardware_match": { "min_gpus": 2, "multi_node": true }, "parallel_flag": "--tensor-parallel-size", "env": { "GLOO_SOCKET_IFNAME": "$IFACE_NAME", "NCCL_SOCKET_IFNAME": "$IFACE_NAME" } }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e ยท 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tp": "/inclusionAI/Ring-1T-FP8/strategies/single_node_tp.json", "multi_node_tp_pp": "/inclusionAI/Ring-1T-FP8/strategies/multi_node_tp_pp.json", "multi_node_tep": "/inclusionAI/Ring-1T-FP8/strategies/multi_node_tep.json", "multi_node_dep": "/inclusionAI/Ring-1T-FP8/strategies/multi_node_dep.json", "pd_cluster": "/inclusionAI/Ring-1T-FP8/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/inclusionAI/Ring-1T-FP8/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/inclusionAI/Ring-1T-FP8/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/inclusionAI/Ring-1T-FP8/hw/h200.json", "h100": "/inclusionAI/Ring-1T-FP8/hw/h100.json", "b200": "/inclusionAI/Ring-1T-FP8/hw/b200.json", "gb200": "/inclusionAI/Ring-1T-FP8/hw/gb200.json", "b300": "/inclusionAI/Ring-1T-FP8/hw/b300.json", "gb300": "/inclusionAI/Ring-1T-FP8/hw/gb300.json", "mi300x": "/inclusionAI/Ring-1T-FP8/hw/mi300x.json", "mi325x": "/inclusionAI/Ring-1T-FP8/hw/mi325x.json", "mi350x": "/inclusionAI/Ring-1T-FP8/hw/mi350x.json", "mi355x": "/inclusionAI/Ring-1T-FP8/hw/mi355x.json" } }, "model": { "model_id": "inclusionAI/Ring-1T-FP8", "min_vllm_version": "0.11.0", "architecture": "moe", "parameter_count": "1T", "active_parameters": "50B", "context_length": 65536, "base_args": [ "--trust-remote-code", "--tensor-parallel-size", "8", "--max-num-seqs", "32", "--kv-cache-dtype", "fp8", "--served-model-name", "Ring-1T-FP8" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": {}, "opt_in_features": [], "variants": { "default": { "precision": "fp8", "vram_minimum_gb": 1200, "description": "FP8 weights on 8x H200 (80 GB) with FP8 KV cache", "extra_args": [ "--gpu-memory-utilization", "0.97", "--compilation-config", "{\"use_inductor\": false}" ] } }, "hardware_overrides": {}, "guide": "## Overview\n\nRing-1T-FP8 is inclusionAI's BailingMoeV2 FP8 model (~1T total parameters). This recipe\ncovers pure tensor-parallel deployment across 8 GPUs on NVIDIA H200 or AMD MI300X+.\n\n## Prerequisites\n\n- Hardware: 8x H200 or 8x MI300X/MI325X/MI355X\n- vLLM >= 0.11.0\n\n### Install vLLM (CUDA)\n\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto\n```\n\n### Install vLLM (AMD ROCm)\n\n```bash\nuv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700\n```\n\n## Launch commands\n\n8x H200 (FP8 KV cache):\n\n```bash\nvllm serve inclusionAI/Ring-1T-FP8 \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --gpu-memory-utilization 0.97 \\\n --max-num-seqs 32 \\\n --kv-cache-dtype fp8 \\\n --compilation-config '{\"use_inductor\": false}' \\\n --served-model-name Ring-1T-FP8\n```\n\n8x MI300X/MI325X/MI355X:\n\n```bash\nexport VLLM_ROCM_USE_AITER=1\nvllm serve inclusionAI/Ring-1T-FP8 \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --gpu-memory-utilization 0.9 \\\n --max-num-seqs 32 \\\n --kv-cache-dtype fp8 \\\n --served-model-name Ring-1T-FP8\n```\n\nTuning flags:\n- `--max-model-len=65536` works well for most scenarios.\n- `--max-num-batched-tokens=32768` for prompt-heavy; 16384/8192 for lower latency.\n- Reduce `--gpu-memory-utilization` below 0.97 if you hit OOM.\n\n## Client Usage\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"Ring-1T-FP8\",\n \"messages\": [{\"role\": \"user\", \"content\": \"9.11 and 9.8, which is greater?\"}]\n }'\n```\n\n## References\n\n- [Ring-1T-FP8 on Hugging Face](https://huggingface.co/inclusionAI/Ring-1T-FP8)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/inclusionAI/Ring-1T-FP8.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/inclusionAI/Ring-1T-FP8.yaml", "overlays": [] } }