{ "hf_id": "inclusionAI/Ling-3.1-flash", "meta": { "title": "Ling-3.1-flash", "slug": "ling-3-1-flash", "provider": "inclusionAI", "description": "Ling-3.1-flash MoE model with block-FP8 weights, tool calling, and reasoning support", "date_added": "2026-10-14", "date_updated": "2026-10-15", "tasks": [ "text" ], "performance_headline": "Block-FP8 weights with BF16 compute and TP8 + EP8 deployment", "related_recipes": [ "inclusionAI/Ling-3.0-flash" ], "hardware": { "mi355x": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tep", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:nightly", "command": "vllm serve inclusionAI/Ling-3.1-flash \\\n --trust-remote-code \\\n --gpu-memory-utilization 0.9 \\\n --enable-chunked-prefill \\\n --compilation-config '{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}' \\\n --enable-prefix-caching \\\n --enable-expert-parallel \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser ling3 \\\n --reasoning-parser ling3", "argv": [ "vllm", "serve", "inclusionAI/Ling-3.1-flash", "--trust-remote-code", "--gpu-memory-utilization", "0.9", "--enable-chunked-prefill", "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}", "--enable-prefix-caching", "--enable-expert-parallel", "--tensor-parallel-size", "8", "--enable-auto-tool-choice", "--tool-call-parser", "ling3", "--reasoning-parser", "ling3" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:nightly inclusionAI/Ling-3.1-flash \\\n --trust-remote-code \\\n --gpu-memory-utilization 0.9 \\\n --enable-chunked-prefill \\\n --compilation-config '{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}' \\\n --enable-prefix-caching \\\n --enable-expert-parallel \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser ling3 \\\n --reasoning-parser ling3", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:nightly", "inclusionAI/Ling-3.1-flash", "--trust-remote-code", "--gpu-memory-utilization", "0.9", "--enable-chunked-prefill", "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}", "--enable-prefix-caching", "--enable-expert-parallel", "--tensor-parallel-size", "8", "--enable-auto-tool-choice", "--tool-call-parser", "ling3", "--reasoning-parser", "ling3" ], "strategy_spec": { "name": "single_node_tep", "deploy_type": "single_node", "display_name": "Tensor + Expert Parallel", "short_label": "TEP", "orientation": "balanced", "description": "Single-node TEP. TP splits dense layers and EP splits expert layers across local GPUs. TP must be set to GPU count to avoid OOM from replicated dense layers. For MoE models only.\n", "hardware_match": { "min_gpus": 2, "max_gpus": 8, "multi_node": false, "architecture": "moe" }, "vllm_args": [ "--enable-expert-parallel" ], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "kv_store_centralized_mooncake": "/inclusionAI/Ling-3.1-flash/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/inclusionAI/Ling-3.1-flash/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/inclusionAI/Ling-3.1-flash/hw/h200.json", "h100": "/inclusionAI/Ling-3.1-flash/hw/h100.json", "b200": "/inclusionAI/Ling-3.1-flash/hw/b200.json", "gb200": "/inclusionAI/Ling-3.1-flash/hw/gb200.json", "b300": "/inclusionAI/Ling-3.1-flash/hw/b300.json", "gb300": "/inclusionAI/Ling-3.1-flash/hw/gb300.json", "mi300x": "/inclusionAI/Ling-3.1-flash/hw/mi300x.json", "mi325x": "/inclusionAI/Ling-3.1-flash/hw/mi325x.json", "mi350x": "/inclusionAI/Ling-3.1-flash/hw/mi350x.json", "mi355x": "/inclusionAI/Ling-3.1-flash/hw/mi355x.json" } }, "model": { "model_id": "inclusionAI/Ling-3.1-flash", "min_vllm_version": "nightly", "nightly_required": true, "docker_image": { "nvidia": "vllm/vllm-openai:nightly", "amd": "vllm/vllm-openai-rocm:nightly" }, "architecture": "moe", "parameter_count": "560B", "active_parameters": "25B", "context_length": 262144, "base_args": [ "--trust-remote-code", "--enable-expert-parallel", "--gpu-memory-utilization", "0.9", "--enable-chunked-prefill", "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}", "--enable-prefix-caching" ], "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --pre \\\n --extra-index-url https://wheels.vllm.ai/nightly/cu130 \\\n --extra-index-url https://download.pytorch.org/whl/cu130 \\\n --index-strategy unsafe-best-match", "note": "vLLM nightly isn't released yet — nightly required. Nightly wheels are CUDA 13 (cu130) only; the host needs an r580+ NVIDIA driver." }, "docker": { "command": "docker pull vllm/vllm-openai:nightly" } } }, "features": { "tool_calling": { "description": "Enable Ling 3 automatic tool calling", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "ling3" ] }, "reasoning": { "description": "Split Ling 3 thinking traces into reasoning", "args": [ "--reasoning-parser", "ling3" ] }, "spec_decoding": { "description": "Speculative decoding with the checkpoint's native MTP head and three speculative tokens", "args": [ "--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":3}" ] } }, "opt_in_features": [ "spec_decoding" ], "variants": { "default": { "precision": "fp8", "vram_minimum_gb": 689, "tp": 8, "description": "Block-FP8 (E4M3) weights, 128x128 block scaling, with BF16 compute." } }, "hardware_overrides": { "amd": { "extra_env": { "VLLM_ROCM_USE_AITER": "1" } } }, "guide": "## Overview\n\n`inclusionAI/Ling-3.1-flash` uses the `BailingMoeV3ForCausalLM`\narchitecture with a hybrid MLA/KDA attention stack, 72 layers, 512 routed\nexperts (8 active per token), and one shared expert. The checkpoint uses\nserialized block-FP8 E4M3 weights with 128x128 block scaling and dynamic\nactivation quantization. Non-quantized modules use BF16.\n\n## Prerequisites\n\n- **vLLM:** nightly build (`vllm/vllm-openai-rocm:nightly` on AMD)\n- **Deployment:** a single node with 8 GPUs, using TP8 + EP8\n- **Precision:** serialized block FP8 weights with BF16 compute\n- **Context length:** 262,144 tokens\n- **Memory:** approximately 689 GB aggregate VRAM as a loading estimate;\n allow additional capacity for KV cache and the intended workload\n\n## Launching the Server\n\n```bash\nvllm serve inclusionAI/Ling-3.1-flash \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --enable-expert-parallel \\\n --gpu-memory-utilization 0.9 \\\n --enable-chunked-prefill \\\n --compilation-config '{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}' \\\n --enable-prefix-caching \\\n --enable-auto-tool-choice \\\n --tool-call-parser ling3 \\\n --reasoning-parser ling3\n```\n\n### AMD MI355X\n\nUse the ROCm nightly image `vllm/vllm-openai-rocm:nightly`,\nenable AITER and launch using the command:\n\n```bash\nVLLM_ROCM_USE_AITER=1 vllm serve inclusionAI/Ling-3.1-flash \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --enable-expert-parallel \\\n --gpu-memory-utilization 0.9 \\\n --enable-chunked-prefill \\\n --compilation-config '{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}' \\\n --enable-prefix-caching \\\n --enable-auto-tool-choice \\\n --tool-call-parser ling3 \\\n --reasoning-parser ling3\n```\n\nEach GPU has about 168 GiB of KV cache available.\n\n## Speculative Decoding (MTP)\n\nThe checkpoint ships a native MTP head. Enable it with three speculative\ntokens:\n\n```bash\n--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'\n```\n\nOn MI355X this gave a mean acceptance length of 3.0 (per-position acceptance\n0.96 / 0.67 / 0.38) on short reasoning prompts.\n\n## Thinking Mode\n\nThinking is selected per request through the chat template rather than by a\nserver flag:\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"EMPTY\")\nresponse = client.chat.completions.create(\n model=\"inclusionAI/Ling-3.1-flash\",\n messages=[{\"role\": \"user\", \"content\": \"Solve the problem step by step.\"}],\n temperature=0.0,\n max_tokens=16384,\n extra_body={\"chat_template_kwargs\": {\"enable_thinking\": True}},\n)\nmsg = response.choices[0].message\nprint(msg.reasoning)\nprint(msg.content)\n```\n\n`--reasoning-parser ling3` separates thinking traces from the final answer.\nKeep the prompt and requested output within the configured context limit.\n\n## References\n\n- [Ling-3.1-flash on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.1-flash)\n- [vLLM nightly installation](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/#install-the-latest-code)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/inclusionAI/Ling-3.1-flash.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/inclusionAI/Ling-3.1-flash.yaml", "overlays": [] } }