# stepfun-ai/Step-3.5-Flash — vLLM serving recipe

> Production-grade reasoning MoE (~196B total / 11B active parameters) with hybrid attention schedules, SWA compensation, and multi-token prediction for low-latency long-context inference

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/stepfun-ai/Step-3.5-Flash
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/stepfun-ai/Step-3.5-Flash.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`, `int4`, `int8`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_tep`, `multi_node_dep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.15.1.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve stepfun-ai/Step-3.5-Flash \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser step3p5 \
  --enable-auto-tool-choice \
  --reasoning-parser step3p5
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest stepfun-ai/Step-3.5-Flash \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser step3p5 \
  --enable-auto-tool-choice \
  --reasoning-parser step3p5
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Step-3.5-Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) is an advanced
reasoning engine from [StepFun](https://www.stepfun.com/company). Highlights:

- Hybrid attention schedules with compensation for sliding-window attention (SWA)
- Sparse MoE structure (196B total parameters, 11B active)
- Multi-token prediction mechanism for faster inference

Available precisions:

- [stepfun-ai/Step-3.5-Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) (BF16)
- [stepfun-ai/Step-3.5-Flash-FP8](https://huggingface.co/stepfun-ai/Step-3.5-Flash-FP8)
- [stepfun-ai/Step-3.5-Flash-Int4](https://huggingface.co/stepfun-ai/Step-3.5-Flash-Int4) (not yet supported by vLLM)

### Prerequisites

- **vLLM version:** latest stable
- **Hardware:** 4x H200/H20/B200

#### Install vLLM

```bash
uv venv
source .venv/bin/activate
uv pip install vllm --torch-backend auto
```

### Launching the Server

#### Tensor Parallel

```bash
vllm serve stepfun-ai/Step-3.5-Flash \
    --tensor-parallel-size 4 \
    --reasoning-parser step3p5 \
    --tool-call-parser step3p5 \
    --enable-auto-tool-choice \
    --trust-remote-code
```

Note: The FP8 version cannot use TP4 — use DP4 instead.

#### Data Parallel + Expert Parallel (recommended for FP8)

```bash
vllm serve stepfun-ai/Step-3.5-Flash \
    --data-parallel-size 4 \
    --enable-expert-parallel \
    --reasoning-parser step3p5 \
    --tool-call-parser step3p5 \
    --enable-auto-tool-choice \
    --trust-remote-code
```

#### Enabling MTP Speculative Decoding

```bash
vllm serve stepfun-ai/Step-3.5-Flash \
    --tensor-parallel-size 4 \
    --reasoning-parser step3p5 \
    --tool-call-parser step3p5 \
    --enable-auto-tool-choice \
    --trust-remote-code \
    --hf-overrides '{"num_nextn_predict_layers": 1}' \
    --speculative-config '{"method": "step3p5_mtp", "num_speculative_tokens": 1}'
```

### Benchmarking

```bash
vllm bench serve \
  --backend vllm \
  --model stepfun-ai/Step-3.5-Flash \
  --endpoint /v1/completions \
  --dataset-name random \
  --random-input 2048 \
  --random-output 1024 \
  --max-concurrency 10 \
  --num-prompt 100
```

### Troubleshooting

- **MoE kernel tuning:** See [tune-moe-kernel](https://github.com/vllm-project/recipes/blob/main/Qwen/Qwen3-Next.md#tune-moe-kernel)
  to tune Triton kernels for your hardware.
- **FP8 DeepGEMM:** For FP8, install DeepGEMM via [install_deepgemm.sh](https://github.com/vllm-project/vllm/blob/v0.16.0rc0/tools/install_deepgemm.sh).
- **B200 FlashInfer FP8 MoE error:** The `routing_logits must be bfloat16`
  error on B200 was fixed in vLLM v0.28.0 — no workaround is needed.
- **FP8 + TP4 incompatibility:** Use DP4+EP instead.

### References

- [Model card](https://huggingface.co/stepfun-ai/Step-3.5-Flash)
- [FP8 checkpoint](https://huggingface.co/stepfun-ai/Step-3.5-Flash-FP8)
- [StepFun](https://www.stepfun.com/company)