# RecursiveSI/flint — vLLM serving recipe

> 39.8B-parameter (2.5B active) MoE with hybrid sliding-window / full attention, reasoning and tool calling; BF16 and NVFP4 checkpoints on a single Blackwell GPU.

Configuration: hardware `gb200` (4× GB200 NVL4) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/RecursiveSI/flint
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/RecursiveSI/flint.json?hardware=gb200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `nvfp4`
- `strategy`: `single_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= nightly.

Use a nightly wheel. The NVFP4 GGUF checkpoint also needs `pip install 'gguf>=0.19.0'`.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve RecursiveSI/flint \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser glm47 \
  --exclude-tools-when-tool-choice-none \
  --reasoning-parser glm47
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest RecursiveSI/flint \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser glm47 \
  --exclude-tools-when-tool-choice-none \
  --reasoning-parser glm47
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Flint is a 39.8B-parameter mixture-of-experts model (256 experts, top-6 sigmoid routing plus a
shared expert; ~2.5B parameters active per token) with 49 layers: 35 sliding-window layers
(window 513, RoPE) and 14 full-attention layers (no RoPE), GQA 32/4, attention sinks, and a
262,144-token context. Two checkpoints are served:

- [RecursiveSI/flint](https://huggingface.co/RecursiveSI/flint): BF16 safetensors.
- [RecursiveSI/flint-nvfp4-gguf](https://huggingface.co/RecursiveSI/flint-nvfp4-gguf): a single
  NVFP4 GGUF (repacked losslessly at load and run as W4A16, like the reference runtime). It ships
  no config or tokenizer; both come from the BF16 repo.

Tested end to end on one GB200 (TP=1, BF16 KV cache, CUDA graphs, no torch.compile) for both
checkpoints. On a single GPU the Flint kernels are always used (fused MoE, small-batch CuTe GEMV,
CuTe decode attention, low-latency decode path for <= 8 requests); TP > 1 and expert parallelism
fall back to stock vLLM layers and are not validated.

### Launch

BF16:

```bash
vllm serve RecursiveSI/flint --served-model-name flint \
  --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47 \
  --exclude-tools-when-tool-choice-none
```

NVFP4 (requires `gguf>=0.19.0`):

```bash
vllm serve RecursiveSI/flint-nvfp4-gguf \
  --hf-config-path RecursiveSI/flint --tokenizer RecursiveSI/flint \
  --served-model-name flint \
  --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47 \
  --exclude-tools-when-tool-choice-none
```

### Client usage

Use `/v1/chat/completions`. Sampling defaults from `generation_config.json`: temperature 1.0,
top_p 0.95, top_k 20 (greedy decoding is not validated). Reasoning is returned in `reasoning`
and counts toward `max_tokens`; use `max_tokens` of at least 32768 for hard problems.

```bash
curl localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "flint",
  "messages": [{"role": "user", "content": "What is 17*23? Answer briefly."}],
  "max_tokens": 1024
}'
```

Accuracy on GB200 with this recipe (BF16 / NVFP4): GSM8K 0.950 / 0.950, GPQA-Diamond (4 repeats)
0.721 / 0.696, MRCR (48 samples, up to 114K-token prompts) ~0.47 / ~0.46, tool-calling suite
(19 cases x 10: typed args, tool selection, parallel calls, multi-turn, tool_choice none,
escaping, streaming) 10/10 on every case.

### Benchmarking

Forward time on one GB200 (`vllm bench forward`, batch x context, us):

| shape | BF16 | NVFP4 |
|---|---|---|
| decode 1 x 100K | 2341 | 1783 |
| decode 8 x 100K | 6836 | 5703 |
| decode 32 x 100K | 19580 | 16716 |
| prefill 8K @ 100K | 145617 | 144765 |

Serving (NVFP4, `vllm bench serve`, random 1000 in / 1000 out): TPOT 2.0 ms at concurrency 1,
2.8 ms at 8, 4.9 ms at 32, 8.5 ms at 128; median inter-token latency 2.06 ms at batch 1 with
100K context. Prefill runs eagerly and is CPU-bound on kernel-launch overhead: TTFT is ~60 ms
for a short prompt and ~125 ms for an 8K-token prompt.

### Troubleshooting

- Rust frontend: not supported for this recipe. It rejects `--exclude-tools-when-tool-choice-none`,
  and for NVFP4 also `--hf-config-path` / `--tokenizer`; keep the default Python frontend.
- Reasoning parser: the reference runtime's `deepseek-v3` name maps to vLLM's `deepseek_v3`
  parser, which only extracts reasoning when the request enables `thinking`; with Flint the
  reasoning then stays in `content`. Use `glm47`.
- If `content` contains literal `<think>` or `<tool_call>` text, the server was started without
  the parsers above.
- Batch-1 decode time varies by ~3-4% between sessions (a GPU-side state, not configuration).

### References

- [Flint (BF16) model card](https://huggingface.co/RecursiveSI/flint)
- [Flint NVFP4 model card](https://huggingface.co/RecursiveSI/flint-nvfp4-gguf)