# black-forest-labs/flux-3-image-klein — vLLM serving recipe

> 5.6B distilled image transformer for text-to-image and image editing, served via vLLM-Omni with a Qwen3-VL-4B text encoder

Configuration: hardware `h200` (8× H200) · variant `default` (BF16) · task `t2i`

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/black-forest-labs/flux-3-image-klein
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/black-forest-labs/flux-3-image-klein.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`
- `variant`: `default`, `fp8r`, `nvfp4`
- `task`: `t2i`, `i2i`

## Install

Requires vLLM >= 0.32.0.

installs vLLM-Omni from source (main) — FLUX 3 Image Klein support ships in the upcoming release

```bash
uv pip install git+https://github.com/vllm-project/vllm-omni.git
```

Extra dependencies:

- Accept the FLUX Kommunity License on Hugging Face first, then authenticate.

```bash
hf auth login
```

## Serve

vLLM-Omni recipe — the generation endpoints and request examples are in the guide below.

```bash
vllm serve black-forest-labs/flux-3-image-klein \
  --port 8099 \
  --omni
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[FLUX 3 Image Klein](https://huggingface.co/black-forest-labs/flux-3-image-klein)
is a 5.6B distilled image transformer from Black Forest Labs for text-to-image
generation and image editing (single- and multi-reference). It uses
Qwen3-VL-4B-Instruct as the text encoder and runs a fixed 16-step schedule
with no classifier-free guidance. vLLM-Omni serves it through an
OpenAI-compatible API:

| Mode | Endpoint | Request body |
|------|----------|--------------|
| T2I | `/v1/images/generations` | JSON (`prompt`, `size`, `seed`) |
| I2I | `/v1/chat/completions` | Chat message with the reference image as a base64 data URI |

The checkpoint is a single package: DiT (`model.safetensors`), image
autoencoder (`ae.safetensors`), and `text_encoder/` at the root, with
quantized DiT packages under `variants/fp8r` and `variants/nvfp4`. Select a
variant with
`--stage-overrides '{"0":{"additional_config":{"variant":"fp8r"}}}'`; omit it
for the BF16 root package.

### Prerequisites

- Accept the [FLUX Kommunity License](https://huggingface.co/black-forest-labs/flux-3-image-klein/blob/main/LICENSE.md) on Hugging Face, then `hf auth login`. The license permits non-commercial use with a commercial-output exception for Qualifying Users.
- Recommended: use the day-0 Docker image `vllm/vllm-omni:flux-3-image-klein`, which ships vLLM-Omni and all dependencies. Bare-metal installs use vLLM-Omni from source (see Install above).

### Launching the server

With the day-0 image:

```bash
docker run --runtime nvidia --gpus all \
  -v ${HF_HOME:-$HOME/.cache/huggingface}:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8099:8099 \
  --ipc=host \
  vllm/vllm-omni:flux-3-image-klein \
  vllm serve black-forest-labs/flux-3-image-klein --omni \
    --served-model-name flux-3-image-klein \
    --port 8099
```

Bare metal:

```bash
vllm serve black-forest-labs/flux-3-image-klein --omni \
  --served-model-name flux-3-image-klein \
  --port 8099

# FP8R / NVFP4 DiT variant
vllm serve black-forest-labs/flux-3-image-klein --omni \
  --served-model-name flux-3-image-klein \
  --stage-overrides '{"0":{"additional_config":{"variant":"fp8r"}}}' \
  --port 8099
```

Useful extras (see `examples/online_serving/flux3/run_server.sh` in the vLLM-Omni repository):

- `--tensor-parallel-size N` / `--cfg-parallel-size N` for multi-GPU scaling.
- `--fp8-attention` (FlashInfer/cuDNN FP8 DiT attention, SM100 only, needs cuDNN >= 9.17.1).
- `--lora-path /path/to/adapter --lora-scale 1.0` to preload a PEFT LoRA adapter; each request still selects the adapter by path.
- `uv pip install -e '.[prims-ts]'` enables PrimsTS kernels for FP8R/NVFP4 packages on SM100/SM103/SM107; BF16 always uses vLLM kernels.

### Offline inference

```bash
# T2I
python examples/offline_inference/flux3/end2end.py \
  --model black-forest-labs/flux-3-image-klein \
  --prompt "a serene alpine meadow at golden hour, wildflowers in the foreground"

# I2I (conditioning image; repeat --image for multi-reference, up to 10)
python examples/offline_inference/flux3/end2end.py \
  --model black-forest-labs/flux-3-image-klein \
  --image /absolute/path/to/reference.png \
  --prompt "restyle as a watercolor painting with soft pastel tones" \
  --seed 7
```

### Prompt upsampling (optional)

[black-forest-labs/flux-3-image-klein-pup](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup)
is a Qwen3.5-9B vision-language fine-tune that expands a short request into
the detailed caption FLUX 3 Image Klein works best with — the reference
inference code runs it in front of the image model by default. It is a
standard `Qwen3_5ForConditionalGeneration` checkpoint, so it is served with
the stock vLLM image on a separate GPU:

```bash
hf download black-forest-labs/flux-3-image-klein-pup --local-dir /path/to/pup

docker run --gpus '"device=0"' --ipc=host -p 8000:8000 \
  -v /path/to/pup:/model:ro \
  vllm/vllm-openai:v0.29.0 \
  --model /model \
  --served-model-name flux-3-image-klein-pup \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --reasoning-parser qwen3
```

The system prompts (`system_prompts.json` in the pup repo) select
text-to-image vs. editing mode, and `t2i_bbox` / `i2i_bbox` variants return
the caption plus a JSON table of `[y_min, x_min, y_max, x_max]` boxes on the
0-1000 grid. A ready-made client is `openai_client_pup.py` next to
`openai_client.py` in `examples/online_serving/flux3/` of the vLLM-Omni
repository — pass `--image` repeatedly (up to 10) for multi-reference
editing; feed its printed caption to the image model as the prompt.

### Behavior and constraints

- **Distilled package**: do not pass `guidance_scale` or a nonempty
  `negative_prompt` — the model rejects them. Step count and flow shift
  default to the package config (16 steps, shift 6.93); `--steps` and
  `--flow-shift` override them.
- **Multi-reference editing**: I2I accepts up to 10 reference images, sent
  in order as multiple `image_url` entries in the chat message. Prompts can
  cite them by position ("the first image", "the second image", ...).
- **I2I has no strength parameter**: the reference image is fed to the DiT as
  conditioning tokens and generation always starts from pure noise. The seed
  controls how far the output drifts from the reference — some seeds stay
  very close to the input, so change the seed when you want a bolder restyle.
- Startup logs show two benign `Error parsing config ... Should have a
  'model_type' key` lines: the package's root `config.json` is a custom FLUX
  format, and vLLM-Omni falls back to the Flux3 pipeline automatically.
- Bounding boxes on a 0-1000 grid can be embedded in the prompt to place,
  move, recolor, or remove elements (see the model card).

### Verified setup

Recipe validated end-to-end on 2026-10-15 with the day-0 image build
(arm64, CUDA 13.0) on a GB200: T2I 1024x1024 in ~2.1s and I2I in ~2.8s
after a ~105s engine startup; ~19 GiB GPU memory with the BF16 package.

### References

- [FLUX 3 Image Klein model card](https://huggingface.co/black-forest-labs/flux-3-image-klein)
- [FLUX 3 Image Klein Base](https://huggingface.co/black-forest-labs/flux-3-image-klein-base) (undistilled, for fine-tuning / LoRA training)
- [FLUX 3 Image Klein PUP](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup) (prompt upsampler)
- [vLLM-Omni](https://github.com/vllm-project/vllm-omni)