{ "hf_id": "RecursiveSI/flint", "meta": { "title": "Flint", "slug": "flint", "provider": "RecursiveSI", "description": "39.8B-parameter (2.5B active) MoE with hybrid sliding-window / full attention, reasoning and tool calling; BF16 and NVFP4 checkpoints on a single Blackwell GPU.", "date_added": "2026-10-18", "date_updated": "2026-10-21", "tasks": [ "text" ], "performance_headline": "Single GB200, NVFP4: ~2.0 ms per output token at batch 1 with 100K context", "related_recipes": [], "default_hardware": "gb200", "hardware": { "gb200": "verified" } }, "recommended_command": { "hardware": "gb200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve RecursiveSI/flint \\\n --tensor-parallel-size 1 \\\n --enable-auto-tool-choice \\\n --tool-call-parser glm47 \\\n --exclude-tools-when-tool-choice-none \\\n --reasoning-parser glm47", "argv": [ "vllm", "serve", "RecursiveSI/flint", "--tensor-parallel-size", "1", "--enable-auto-tool-choice", "--tool-call-parser", "glm47", "--exclude-tools-when-tool-choice-none", "--reasoning-parser", "glm47" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest RecursiveSI/flint \\\n --tensor-parallel-size 1 \\\n --enable-auto-tool-choice \\\n --tool-call-parser glm47 \\\n --exclude-tools-when-tool-choice-none \\\n --reasoning-parser glm47", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "RecursiveSI/flint", "--tensor-parallel-size", "1", "--enable-auto-tool-choice", "--tool-call-parser", "glm47", "--exclude-tools-when-tool-choice-none", "--reasoning-parser", "glm47" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "blackwell", "compute_arch": "sm100", "display_name": "GB200 NVL4", "description": "NVIDIA GB200 Grace-Blackwell compute tray · 192 GB HBM3e/GPU · 4 Blackwell GPUs (NVL72 tray unit)", "gpu_count": 4, "vram_gb": 768, "multi_node": false }, "alternatives": { "kv_store_centralized_mooncake": "/RecursiveSI/flint/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/RecursiveSI/flint/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "gb200": "/RecursiveSI/flint/hw/gb200.json", "h100": "/RecursiveSI/flint/hw/h100.json", "h200": "/RecursiveSI/flint/hw/h200.json", "b200": "/RecursiveSI/flint/hw/b200.json", "b300": "/RecursiveSI/flint/hw/b300.json", "gb300": "/RecursiveSI/flint/hw/gb300.json", "mi300x": "/RecursiveSI/flint/hw/mi300x.json", "mi325x": "/RecursiveSI/flint/hw/mi325x.json", "mi350x": "/RecursiveSI/flint/hw/mi350x.json", "mi355x": "/RecursiveSI/flint/hw/mi355x.json" } }, "model": { "model_id": "RecursiveSI/flint", "min_vllm_version": "nightly", "architecture": "moe", "parameter_count": "39.8B", "active_parameters": "2.5B", "context_length": 262144, "base_args": [], "base_env": {}, "rust_frontend": "unsupported", "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto", "note": "Use a nightly wheel. The NVFP4 GGUF checkpoint also needs `pip install 'gguf>=0.19.0'`." }, "docker": { "command": "docker pull vllm/vllm-openai:latest", "note": "Use the vLLM nightly image." } } }, "features": { "tool_calling": { "description": "Automatic tool choice with the GLM-4.7 XML tool-call parser (Flint's chat template emits //); tools are dropped from the prompt when tool_choice is none, as in the reference runtime.", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "glm47", "--exclude-tools-when-tool-choice-none" ] }, "reasoning": { "description": "Return the … block in the reasoning field; the model decides per turn whether to reason.", "args": [ "--reasoning-parser", "glm47" ] } }, "opt_in_features": [], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 96, "description": "BF16 safetensors (79.6 GB of weights)." }, "nvfp4": { "model_id": "RecursiveSI/flint-nvfp4-gguf", "precision": "nvfp4", "vram_minimum_gb": 29, "description": "NVFP4 GGUF served weight-only (W4A16); 23.5 GB resident after load (the LM head is dequantized to BF16). Config and tokenizer come from the BF16 repo.", "extra_args": [ "--hf-config-path", "RecursiveSI/flint", "--tokenizer", "RecursiveSI/flint" ], "json": "/RecursiveSI/flint-nvfp4-gguf.json" } }, "hardware_overrides": {}, "guide": "## Overview\n\nFlint is a 39.8B-parameter mixture-of-experts model (256 experts, top-6 sigmoid routing plus a\nshared expert; ~2.5B parameters active per token) with 49 layers: 35 sliding-window layers\n(window 513, RoPE) and 14 full-attention layers (no RoPE), GQA 32/4, attention sinks, and a\n262,144-token context. Two checkpoints are served:\n\n- [RecursiveSI/flint](https://huggingface.co/RecursiveSI/flint): BF16 safetensors.\n- [RecursiveSI/flint-nvfp4-gguf](https://huggingface.co/RecursiveSI/flint-nvfp4-gguf): a single\n NVFP4 GGUF (repacked losslessly at load and run as W4A16, like the reference runtime). It ships\n no config or tokenizer; both come from the BF16 repo.\n\nTested end to end on one GB200 (TP=1, BF16 KV cache, CUDA graphs, no torch.compile) for both\ncheckpoints. On a single GPU the Flint kernels are always used (fused MoE, small-batch CuTe GEMV,\nCuTe decode attention, low-latency decode path for <= 8 requests); TP > 1 and expert parallelism\nfall back to stock vLLM layers and are not validated.\n\n## Launch\n\nBF16:\n\n```bash\nvllm serve RecursiveSI/flint --served-model-name flint \\\n --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47 \\\n --exclude-tools-when-tool-choice-none\n```\n\nNVFP4 (requires `gguf>=0.19.0`):\n\n```bash\nvllm serve RecursiveSI/flint-nvfp4-gguf \\\n --hf-config-path RecursiveSI/flint --tokenizer RecursiveSI/flint \\\n --served-model-name flint \\\n --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47 \\\n --exclude-tools-when-tool-choice-none\n```\n\n## Client usage\n\nUse `/v1/chat/completions`. Sampling defaults from `generation_config.json`: temperature 1.0,\ntop_p 0.95, top_k 20 (greedy decoding is not validated). Reasoning is returned in `reasoning`\nand counts toward `max_tokens`; use `max_tokens` of at least 32768 for hard problems.\n\n```bash\ncurl localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{\n \"model\": \"flint\",\n \"messages\": [{\"role\": \"user\", \"content\": \"What is 17*23? Answer briefly.\"}],\n \"max_tokens\": 1024\n}'\n```\n\nAccuracy on GB200 with this recipe (BF16 / NVFP4): GSM8K 0.950 / 0.950, GPQA-Diamond (4 repeats)\n0.721 / 0.696, MRCR (48 samples, up to 114K-token prompts) ~0.47 / ~0.46, tool-calling suite\n(19 cases x 10: typed args, tool selection, parallel calls, multi-turn, tool_choice none,\nescaping, streaming) 10/10 on every case.\n\n## Benchmarking\n\nForward time on one GB200 (`vllm bench forward`, batch x context, us):\n\n| shape | BF16 | NVFP4 |\n|---|---|---|\n| decode 1 x 100K | 2341 | 1783 |\n| decode 8 x 100K | 6836 | 5703 |\n| decode 32 x 100K | 19580 | 16716 |\n| prefill 8K @ 100K | 145617 | 144765 |\n\nServing (NVFP4, `vllm bench serve`, random 1000 in / 1000 out): TPOT 2.0 ms at concurrency 1,\n2.8 ms at 8, 4.9 ms at 32, 8.5 ms at 128; median inter-token latency 2.06 ms at batch 1 with\n100K context. Prefill runs eagerly and is CPU-bound on kernel-launch overhead: TTFT is ~60 ms\nfor a short prompt and ~125 ms for an 8K-token prompt.\n\n## Troubleshooting\n\n- Rust frontend: not supported for this recipe. It rejects `--exclude-tools-when-tool-choice-none`,\n and for NVFP4 also `--hf-config-path` / `--tokenizer`; keep the default Python frontend.\n- Reasoning parser: the reference runtime's `deepseek-v3` name maps to vLLM's `deepseek_v3`\n parser, which only extracts reasoning when the request enables `thinking`; with Flint the\n reasoning then stays in `content`. Use `glm47`.\n- If `content` contains literal `` or `` text, the server was started without\n the parsers above.\n- Batch-1 decode time varies by ~3-4% between sessions (a GPU-side state, not configuration).\n\n## References\n\n- [Flint (BF16) model card](https://huggingface.co/RecursiveSI/flint)\n- [Flint NVFP4 model card](https://huggingface.co/RecursiveSI/flint-nvfp4-gguf)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/RecursiveSI/flint.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/RecursiveSI/flint.yaml", "overlays": [] } }