# nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 — vLLM serving recipe

> NVIDIA Nemotron-3-Nano 4B (Mamba-hybrid dense) — compact reasoning + tool-use model with BF16 and FP8 variants

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`, `arc_pro_b60`, `arc_pro_b70`
- `variant`: `default`, `fp8`
- `strategy`: `single_node_tp`, `multi_node_tp`, `multi_node_tp_pp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.11.2.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \
  --trust-remote-code \
  --async-scheduling \
  --kv-cache-dtype auto \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser-plugin nano_v3_reasoning_parser.py \
  --reasoning-parser nano_v3
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \
  --trust-remote-code \
  --async-scheduling \
  --kv-cache-dtype auto \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser-plugin nano_v3_reasoning_parser.py \
  --reasoning-parser nano_v3
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Nemotron-3-Nano-4B is the smallest member of the Nemotron-3 hybrid-Mamba family.
It's tuned for low-latency reasoning, tool calling, and edge deployments — DGX Spark
and Jetson Thor are both supported alongside standard Hopper/Blackwell servers.

### Prerequisites

- Hardware: 1x H100/H200/B200, DGX Spark, Jetson Thor, or Intel Arc Pro B60/B70
- vLLM >= 0.11.2 (0.28.0 recommended)
- Docker with NVIDIA Container Toolkit (recommended)

#### Pull Docker Image

```bash
docker pull --platform linux/amd64 vllm/vllm-openai:v0.28.0
```

Jetson Thor:

```bash
docker pull ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor
```

Intel XPU (B60 / B70):

```bash
docker pull vllm/vllm-openai-xpu:latest
```

### Launch commands

BF16:

```bash
wget https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16/resolve/main/nano_v3_reasoning_parser.py

vllm serve nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \
  --trust-remote-code \
  --async-scheduling \
  --max-model-len 262144 \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser-plugin nano_v3_reasoning_parser.py \
  --reasoning-parser nano_v3
```

FP8:

```bash
vllm serve nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --trust-remote-code \
  --async-scheduling \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 1
```

#### Intel XPU (B60, B70)

Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.

BF16:
```bash
docker run --device /dev/dri \
  -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-xpu:latest \
  nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \
  --trust-remote-code \
  --async-scheduling \
  --kv-cache-dtype auto \
  --tensor-parallel-size 1 \
  --enforce-eager \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser-plugin nano_v3_reasoning_parser.py \
  --reasoning-parser nano_v3 \
  --max-model-len 8192
```

FP8:
```bash
docker run --device /dev/dri \
  -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-xpu:latest \
  nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --trust-remote-code \
  --async-scheduling \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 1 \
  --enforce-eager \
  --max-model-len 8192
```

### Benchmarking

```bash
vllm bench serve \
  --model nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \
  --trust-remote-code \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 1024 \
  --num-warmups 20 \
  --ignore-eos \
  --max-concurrency 256 \
  --num-prompts 1024
```

### References

- [Nemotron-3-Nano-4B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16)
- [Nemotron-3-Nano-4B-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8)