# internlm/Intern-S2-397B — vLLM serving recipe

> Flagship scientific multimodal MoE (397B total / 17B active) on the Qwen3.5 hybrid linear/full-attention architecture — 262K context, MTP-accelerated reasoning. BF16 and FP8 checkpoints.

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`, `encoder_parallel`

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/internlm/Intern-S2-397B
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/internlm/Intern-S2-397B.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_dep`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding`, `text_only`, `encoder_parallel`, `long_context` (default: `tool_calling`, `reasoning`, `encoder_parallel`)

## Install

Requires vLLM >= 0.22.1.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
export VLLM_DEEP_GEMM_WARMUP=skip
export VLLM_USE_DEEP_GEMM=0
vllm serve internlm/Intern-S2-397B \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --moe-backend flashinfer_trtllm \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_DEEP_GEMM_WARMUP=skip \
  -e VLLM_USE_DEEP_GEMM=0 \
  vllm/vllm-openai:latest internlm/Intern-S2-397B \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --moe-backend flashinfer_trtllm \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Intern-S2-397B](https://huggingface.co/internlm/Intern-S2-397B) is Shanghai
AI Laboratory's flagship multimodal foundation model for scientific
intelligence and long-horizon agents. It packs 397B total / 17B active
parameters across 512 experts on the Qwen3.5 MoE architecture
(`Qwen3_5MoeForConditionalGeneration`) with hybrid linear/full attention, a
262K context window, and a built-in shared-weight MTP head for speculative
decoding. An official FP8 checkpoint
([internlm/Intern-S2-397B-FP8](https://huggingface.co/internlm/Intern-S2-397B-FP8))
halves the memory footprint and is the configuration the vendor validates.

Beyond chat and reasoning, it handles image and video inputs and improves
agent capabilities for scientific workflows. Time-series inference exists but
is currently LMDeploy-only (see Troubleshooting).

### Prerequisites

- **vLLM version:** 0.22.1 or newer (per the official deployment guide)
- **Hardware (FP8, recommended):** 8x H100 or 8x H200
- **Hardware (BF16):** 8x H200 (141 GB each)
- **Trust remote code:** required — the repo ships a custom tokenizer
  (`InternS1Tokenizer`)
- **MoE kernel env:** the deployment guide disables DeepGEMM
  (`VLLM_USE_DEEP_GEMM=0`, already set by this recipe) and selects the
  FlashInfer TRTLLM MoE backend on NVIDIA (applied automatically when you
  pick an NVIDIA GPU above)

#### Install vLLM

```bash
uv venv
source .venv/bin/activate
uv pip install -U "vllm>=0.22.1" --torch-backend=auto
```

### Launch commands

#### Recommended — official FP8 checkpoint, TP=8

```bash
vllm serve internlm/Intern-S2-397B-FP8 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --mm-encoder-tp-mode data
```

#### BF16 (8x H200)

```bash
vllm serve internlm/Intern-S2-397B \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --mm-encoder-tp-mode data
```

#### With MTP speculative decoding

```bash
vllm serve internlm/Intern-S2-397B-FP8 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --mm-encoder-tp-mode data \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```

#### Long-context (YaRN, up to ~1M)

The base config sets `max_position_embeddings = 262144`. For longer contexts,
turn on the **Long Context** feature above — it applies the deployment
guide's YaRN RoPE override (factor 4.0) and raises `--max-model-len` to
1,010,000. Static YaRN holds the scaling factor constant regardless of input
length, so leave it off unless you actually serve long prompts; for a ~524K
workload, halve the `factor` to `2.0`.

### Client Usage

Recommended sampling parameters from the model card:

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="internlm/Intern-S2-397B-FP8",
    messages=[{"role": "user", "content": "Design a synthesis route for paracetamol."}],
    temperature=0.8,
    top_p=0.95,
    max_tokens=32768,
    extra_body={
        "top_k": 50,
        "min_p": 0.0,
        "spaces_between_special_tokens": False,
    },
)
print(resp.choices[0].message.content)
```

#### Toggle thinking mode

Thinking is enabled by default. Disable it per request:

```python
resp = client.chat.completions.create(
    model="internlm/Intern-S2-397B-FP8",
    messages=[{"role": "user", "content": "What is AGI?"}],
    temperature=0.8,
    top_p=0.95,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
```

> The model card notes: do **not** disable thinking mode for agentic tasks.

### Troubleshooting

- **OOM at full 262K context:** drop `--max-model-len` to 65536 or 131072,
  or lower `--gpu-memory-utilization` headroom.
- **DeepGEMM warmup failures / slow startup:** this recipe already exports
  `VLLM_DEEP_GEMM_WARMUP=skip` and `VLLM_USE_DEEP_GEMM=0` per the deployment
  guide — if you launch outside the builder, set them yourself.
- **Time-series inputs** (`time_series_url`): currently supported only on
  LMDeploy, not vLLM — see the model card.
- **Wrong parser name:** LMDeploy examples use `--tool-call-parser
  interns2-preview`; on vLLM the correct parser is `qwen3_coder` (already
  wired into the Tool Calling feature above).

### References

- [Model card](https://huggingface.co/internlm/Intern-S2-397B)
- [FP8 checkpoint](https://huggingface.co/internlm/Intern-S2-397B-FP8)
- [Deployment guide](https://huggingface.co/internlm/Intern-S2-397B-FP8/blob/main/deployment_guide.md)
- [Intern-S1 GitHub](https://github.com/InternLM/Intern-S1)
- [vLLM documentation](https://docs.vllm.ai/)