{ "hf_id": "internlm/Intern-S2-397B", "meta": { "title": "Intern-S2-397B", "slug": "intern-s2-397b", "provider": "InternLM", "description": "Flagship scientific multimodal MoE (397B total / 17B active) on the Qwen3.5 hybrid linear/full-attention architecture — 262K context, MTP-accelerated reasoning. BF16 and FP8 checkpoints.", "date_added": "2026-09-14", "date_updated": "2026-09-14", "tasks": [ "multimodal", "text" ], "performance_headline": "397B-A17B scientific multimodal MoE — FP8 on 8x H100/H200 per the official deployment guide", "related_recipes": [ "internlm/Intern-S2-Preview", "internlm/Intern-S1", "Qwen/Qwen3.5-397B-A17B" ] }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": { "VLLM_DEEP_GEMM_WARMUP": "skip", "VLLM_USE_DEEP_GEMM": "0" }, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve internlm/Intern-S2-397B \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --moe-backend flashinfer_trtllm \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3 \\\n --mm-encoder-tp-mode data", "argv": [ "vllm", "serve", "internlm/Intern-S2-397B", "--trust-remote-code", "--tensor-parallel-size", "8", "--moe-backend", "flashinfer_trtllm", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "qwen3", "--mm-encoder-tp-mode", "data" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_DEEP_GEMM_WARMUP=skip \\\n -e VLLM_USE_DEEP_GEMM=0 \\\n vllm/vllm-openai:latest internlm/Intern-S2-397B \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --moe-backend flashinfer_trtllm \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3 \\\n --mm-encoder-tp-mode data", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_DEEP_GEMM_WARMUP=skip", "-e", "VLLM_USE_DEEP_GEMM=0", "vllm/vllm-openai:latest", "internlm/Intern-S2-397B", "--trust-remote-code", "--tensor-parallel-size", "8", "--moe-backend", "flashinfer_trtllm", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "qwen3", "--mm-encoder-tp-mode", "data" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tep": "/internlm/Intern-S2-397B/strategies/single_node_tep.json", "single_node_dep": "/internlm/Intern-S2-397B/strategies/single_node_dep.json", "multi_node_tp": "/internlm/Intern-S2-397B/strategies/multi_node_tp.json", "multi_node_tp_pp": "/internlm/Intern-S2-397B/strategies/multi_node_tp_pp.json", "multi_node_dep": "/internlm/Intern-S2-397B/strategies/multi_node_dep.json", "multi_node_tep": "/internlm/Intern-S2-397B/strategies/multi_node_tep.json", "pd_cluster": "/internlm/Intern-S2-397B/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/internlm/Intern-S2-397B/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/internlm/Intern-S2-397B/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/internlm/Intern-S2-397B/hw/h200.json", "h100": "/internlm/Intern-S2-397B/hw/h100.json", "b200": "/internlm/Intern-S2-397B/hw/b200.json", "gb200": "/internlm/Intern-S2-397B/hw/gb200.json", "b300": "/internlm/Intern-S2-397B/hw/b300.json", "gb300": "/internlm/Intern-S2-397B/hw/gb300.json", "mi300x": "/internlm/Intern-S2-397B/hw/mi300x.json", "mi325x": "/internlm/Intern-S2-397B/hw/mi325x.json", "mi350x": "/internlm/Intern-S2-397B/hw/mi350x.json", "mi355x": "/internlm/Intern-S2-397B/hw/mi355x.json" } }, "model": { "model_id": "internlm/Intern-S2-397B", "min_vllm_version": "0.22.1", "architecture": "moe", "parameter_count": "397B", "active_parameters": "17B", "context_length": 262144, "base_args": [ "--trust-remote-code" ], "base_env": { "VLLM_DEEP_GEMM_WARMUP": "skip", "VLLM_USE_DEEP_GEMM": "0" }, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "tool_calling": { "description": "Enable automatic tool choice with the Qwen3 Coder parser (per official deployment guide)", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder" ] }, "reasoning": { "description": "Extract ... chain-of-thought via the Qwen3 reasoning parser", "args": [ "--reasoning-parser", "qwen3" ] }, "spec_decoding": { "description": "Shared-weight MTP speculative decoding — 3 draft tokens (per official deployment guide)", "args": [ "--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":3}" ] }, "text_only": { "description": "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel.", "args": [ "--language-model-only" ] }, "encoder_parallel": { "description": "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only.", "args": [ "--mm-encoder-tp-mode", "data" ] }, "long_context": { "description": "YaRN RoPE scaling past the native 262,144-token window to ~1,010,000 tokens (per official deployment guide). Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0.", "args": [ "--hf-overrides", "{\"text_config\": {\"rope_parameters\": {\"mrope_interleaved\": true, \"mrope_section\": [11, 11, 10], \"rope_type\": \"yarn\", \"rope_theta\": 10000000, \"partial_rotary_factor\": 0.25, \"factor\": 4.0, \"original_max_position_embeddings\": 262144}}}", "--max-model-len", "1010000" ], "env": { "VLLM_ALLOW_LONG_MAX_MODEL_LEN": "1" } } }, "opt_in_features": [ "spec_decoding", "text_only", "long_context" ], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 969, "description": "Full precision BF16 — requires 8x H200 or equivalent" }, "fp8": { "model_id": "internlm/Intern-S2-397B-FP8", "precision": "fp8", "vram_minimum_gb": 488, "description": "Official FP8 checkpoint — the configuration validated in the deployment guide on 8x H100/H200", "json": "/internlm/Intern-S2-397B-FP8.json" } }, "hardware_overrides": { "nvidia": { "extra_args": [ "--moe-backend", "flashinfer_trtllm" ] } }, "guide": "## Overview\n\n[Intern-S2-397B](https://huggingface.co/internlm/Intern-S2-397B) is Shanghai\nAI Laboratory's flagship multimodal foundation model for scientific\nintelligence and long-horizon agents. It packs 397B total / 17B active\nparameters across 512 experts on the Qwen3.5 MoE architecture\n(`Qwen3_5MoeForConditionalGeneration`) with hybrid linear/full attention, a\n262K context window, and a built-in shared-weight MTP head for speculative\ndecoding. An official FP8 checkpoint\n([internlm/Intern-S2-397B-FP8](https://huggingface.co/internlm/Intern-S2-397B-FP8))\nhalves the memory footprint and is the configuration the vendor validates.\n\nBeyond chat and reasoning, it handles image and video inputs and improves\nagent capabilities for scientific workflows. Time-series inference exists but\nis currently LMDeploy-only (see Troubleshooting).\n\n## Prerequisites\n\n- **vLLM version:** 0.22.1 or newer (per the official deployment guide)\n- **Hardware (FP8, recommended):** 8x H100 or 8x H200\n- **Hardware (BF16):** 8x H200 (141 GB each)\n- **Trust remote code:** required — the repo ships a custom tokenizer\n (`InternS1Tokenizer`)\n- **MoE kernel env:** the deployment guide disables DeepGEMM\n (`VLLM_USE_DEEP_GEMM=0`, already set by this recipe) and selects the\n FlashInfer TRTLLM MoE backend on NVIDIA (applied automatically when you\n pick an NVIDIA GPU above)\n\n### Install vLLM\n\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U \"vllm>=0.22.1\" --torch-backend=auto\n```\n\n## Launch commands\n\n### Recommended — official FP8 checkpoint, TP=8\n\n```bash\nvllm serve internlm/Intern-S2-397B-FP8 \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --reasoning-parser qwen3 \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --mm-encoder-tp-mode data\n```\n\n### BF16 (8x H200)\n\n```bash\nvllm serve internlm/Intern-S2-397B \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --reasoning-parser qwen3 \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --mm-encoder-tp-mode data\n```\n\n### With MTP speculative decoding\n\n```bash\nvllm serve internlm/Intern-S2-397B-FP8 \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --reasoning-parser qwen3 \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --mm-encoder-tp-mode data \\\n --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'\n```\n\n### Long-context (YaRN, up to ~1M)\n\nThe base config sets `max_position_embeddings = 262144`. For longer contexts,\nturn on the **Long Context** feature above — it applies the deployment\nguide's YaRN RoPE override (factor 4.0) and raises `--max-model-len` to\n1,010,000. Static YaRN holds the scaling factor constant regardless of input\nlength, so leave it off unless you actually serve long prompts; for a ~524K\nworkload, halve the `factor` to `2.0`.\n\n## Client Usage\n\nRecommended sampling parameters from the model card:\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\nresp = client.chat.completions.create(\n model=\"internlm/Intern-S2-397B-FP8\",\n messages=[{\"role\": \"user\", \"content\": \"Design a synthesis route for paracetamol.\"}],\n temperature=0.8,\n top_p=0.95,\n max_tokens=32768,\n extra_body={\n \"top_k\": 50,\n \"min_p\": 0.0,\n \"spaces_between_special_tokens\": False,\n },\n)\nprint(resp.choices[0].message.content)\n```\n\n### Toggle thinking mode\n\nThinking is enabled by default. Disable it per request:\n\n```python\nresp = client.chat.completions.create(\n model=\"internlm/Intern-S2-397B-FP8\",\n messages=[{\"role\": \"user\", \"content\": \"What is AGI?\"}],\n temperature=0.8,\n top_p=0.95,\n extra_body={\"chat_template_kwargs\": {\"enable_thinking\": False}},\n)\n```\n\n> The model card notes: do **not** disable thinking mode for agentic tasks.\n\n## Troubleshooting\n\n- **OOM at full 262K context:** drop `--max-model-len` to 65536 or 131072,\n or lower `--gpu-memory-utilization` headroom.\n- **DeepGEMM warmup failures / slow startup:** this recipe already exports\n `VLLM_DEEP_GEMM_WARMUP=skip` and `VLLM_USE_DEEP_GEMM=0` per the deployment\n guide — if you launch outside the builder, set them yourself.\n- **Time-series inputs** (`time_series_url`): currently supported only on\n LMDeploy, not vLLM — see the model card.\n- **Wrong parser name:** LMDeploy examples use `--tool-call-parser\n interns2-preview`; on vLLM the correct parser is `qwen3_coder` (already\n wired into the Tool Calling feature above).\n\n## References\n\n- [Model card](https://huggingface.co/internlm/Intern-S2-397B)\n- [FP8 checkpoint](https://huggingface.co/internlm/Intern-S2-397B-FP8)\n- [Deployment guide](https://huggingface.co/internlm/Intern-S2-397B-FP8/blob/main/deployment_guide.md)\n- [Intern-S1 GitHub](https://github.com/InternLM/Intern-S1)\n- [vLLM documentation](https://docs.vllm.ai/)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/internlm/Intern-S2-397B.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/internlm/Intern-S2-397B.yaml", "overlays": [] } }