{ "hf_id": "mistralai/Voxtral-Mini-4B-Realtime-2602", "meta": { "title": "Voxtral-Mini-4B-Realtime-2602", "slug": "voxtral-mini-4b-realtime-2602", "provider": "Mistral AI", "description": "Multilingual realtime speech transcription (13 languages) with a natively streaming causal audio encoder; configurable 80ms–2.4s transcription delay served via vLLM's Realtime API", "date_added": "2026-05-13", "date_updated": "2026-05-13", "tasks": [ "multimodal" ], "performance_headline": "Matches offline open-source ASR accuracy at 480ms delay; >12.5 tok/s on a single 16GB GPU", "related_recipes": [], "hardware": { "dgx_spark_gb10": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": { "VLLM_DISABLE_COMPILE_CACHE": "1" }, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve mistralai/Voxtral-Mini-4B-Realtime-2602 \\\n --tokenizer-mode mistral \\\n --compilation_config '{\"cudagraph_mode\": \"PIECEWISE\"}' \\\n --tensor-parallel-size 1", "argv": [ "vllm", "serve", "mistralai/Voxtral-Mini-4B-Realtime-2602", "--tokenizer-mode", "mistral", "--compilation_config", "{\"cudagraph_mode\": \"PIECEWISE\"}", "--tensor-parallel-size", "1" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_DISABLE_COMPILE_CACHE=1 \\\n vllm/vllm-openai:latest mistralai/Voxtral-Mini-4B-Realtime-2602 \\\n --tokenizer-mode mistral \\\n --compilation_config '{\"cudagraph_mode\": \"PIECEWISE\"}' \\\n --tensor-parallel-size 1", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_DISABLE_COMPILE_CACHE=1", "vllm/vllm-openai:latest", "mistralai/Voxtral-Mini-4B-Realtime-2602", "--tokenizer-mode", "mistral", "--compilation_config", "{\"cudagraph_mode\": \"PIECEWISE\"}", "--tensor-parallel-size", "1" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "multi_node_tp": "/mistralai/Voxtral-Mini-4B-Realtime-2602/strategies/multi_node_tp.json", "kv_store_centralized_mooncake": "/mistralai/Voxtral-Mini-4B-Realtime-2602/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/mistralai/Voxtral-Mini-4B-Realtime-2602/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/mistralai/Voxtral-Mini-4B-Realtime-2602/hw/h200.json", "h100": "/mistralai/Voxtral-Mini-4B-Realtime-2602/hw/h100.json", "b200": "/mistralai/Voxtral-Mini-4B-Realtime-2602/hw/b200.json", "gb200": "/mistralai/Voxtral-Mini-4B-Realtime-2602/hw/gb200.json", "b300": "/mistralai/Voxtral-Mini-4B-Realtime-2602/hw/b300.json", "gb300": "/mistralai/Voxtral-Mini-4B-Realtime-2602/hw/gb300.json", "dgx_spark_gb10": "/mistralai/Voxtral-Mini-4B-Realtime-2602/hw/dgx_spark_gb10.json", "mi300x": "/mistralai/Voxtral-Mini-4B-Realtime-2602/hw/mi300x.json", "mi325x": "/mistralai/Voxtral-Mini-4B-Realtime-2602/hw/mi325x.json", "mi350x": "/mistralai/Voxtral-Mini-4B-Realtime-2602/hw/mi350x.json", "mi355x": "/mistralai/Voxtral-Mini-4B-Realtime-2602/hw/mi355x.json" } }, "model": { "model_id": "mistralai/Voxtral-Mini-4B-Realtime-2602", "min_vllm_version": "0.20.0", "architecture": "dense", "parameter_count": "4.4B", "active_parameters": "4.4B", "context_length": 131072, "base_args": [ "--tokenizer-mode", "mistral", "--compilation_config", "{\"cudagraph_mode\": \"PIECEWISE\"}" ], "base_env": { "VLLM_DISABLE_COMPILE_CACHE": "1" }, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" }, "extras": [ { "note": "mistral-common's audio extras bundle soxr/librosa/soundfile plus the Voxtral Realtime tokenizer (>= 1.9.0)", "command": "uv pip install -U \"mistral-common[audio]>=1.9.0\"" }, { "note": "Transformers v5 silences a barrage of warnings emitted when serving Voxtral on v4 (see vllm-project/vllm#34642)", "command": "uv pip install -U transformers" } ] } }, "features": {}, "opt_in_features": [], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 16, "description": "BF16 weights — single 16GB+ GPU, full 131072 (~3h audio) context" } }, "hardware_overrides": {}, "guide": "## Overview\n\nVoxtral Mini 4B Realtime is Mistral AI's multilingual realtime speech\ntranscription model — among the first open-source ASR systems to hit\naccuracy comparable to offline models with a **<500ms** end-to-end delay.\n\n- **Architecture**: ≈3.4B Mistral text LM + ≈970M custom **causal** audio\n encoder, both using sliding-window attention to support \"infinite\" streaming.\n- **Languages**: 13 (English, French, Spanish, German, Russian, Chinese,\n Japanese, Italian, Portuguese, Dutch, Arabic, Hindi, Korean).\n- **Configurable delay**: any multiple of 80ms between 80ms and 1200ms,\n plus 2400ms as a standalone value. Default is **480ms**, the sweet spot\n Mistral identified between latency and accuracy.\n- **Context**: 131072 tokens (≈3h of audio at 80ms/token).\n\n## Prerequisites\n\n- **Hardware**: a single GPU with **≥ 16 GB** VRAM (BF16 weights only).\n- **vLLM**: **>= 0.20.0** — the Voxtral Realtime architecture has been\n registered since v0.16.0, but v0.20.0 is the first stable release with\n the architecture documented in the supported-models list.\n\n### Install vLLM and audio dependencies\n\n```bash\nuv venv && source .venv/bin/activate\nuv pip install -U vllm --torch-backend=auto\nuv pip install -U \"mistral-common[audio]>=1.9.0\" transformers\n```\n\nVerify the audio extras pulled `mistral_common >= 1.9.0`:\n\n```bash\npython -c \"import mistral_common; print(mistral_common.__version__)\"\n```\n\n## Launch command\n\n```bash\nVLLM_DISABLE_COMPILE_CACHE=1 vllm serve mistralai/Voxtral-Mini-4B-Realtime-2602 \\\n --tokenizer-mode mistral \\\n --compilation_config '{\"cudagraph_mode\": \"PIECEWISE\"}'\n```\n\n> `--tokenizer-mode mistral` is **required**: Voxtral Realtime's tokenizer\n> only loads through `mistral_common`. Omitting it raises a tokenizer\n> initialization error at startup.\n\nOnce it starts you should see the Realtime API route registered:\n\n```\nRoute: /v1/realtime, Endpoint: realtime_endpoint\n```\n\n### Tuning flags\n\n- `--max-num-batched-tokens` — balance throughput vs latency (higher means\n more throughput at the cost of per-request latency).\n- `--max-model-len` — defaults to 131072 (≈3h). Reduce it if you know your\n sessions are shorter; this cuts the memory reserved for pre-computed\n RoPE frequencies. As a rule of thumb, one text token ≈ 80ms of audio,\n so a 1h meeting needs `--max-model-len >= 3600/0.08 = 45000`.\n\n## Client usage\n\n### Recommended settings\n\n- Always set `temperature=0.0`.\n- Use **WebSockets** against `/v1/realtime` for streaming audio sessions.\n- Adjust the transcription delay by editing the `transcription_delay_ms`\n field in the model's\n [`tekken.json`](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602/blob/main/tekken.json)\n to any multiple of 80ms in `[80, 1200]`, or to `2400`.\n\n### Stream an audio file\n\nSee vLLM's\n[Realtime audio file client example](https://docs.vllm.ai/en/latest/examples/online_serving/openai_realtime_client/).\n\n### Live microphone (Gradio demo)\n\nSee vLLM's\n[Realtime microphone client example](https://docs.vllm.ai/en/latest/examples/online_serving/openai_realtime_microphone_client/)\nfor an end-to-end live-transcription UI.\n\n## Benchmarks (Fleurs, average WER)\n\n| Delay | AVG | English | French | Chinese | Japanese |\n|---------|--------|---------|--------|---------|----------|\n| 160ms | 12.60% | 6.46% | 9.75% | 17.67% | 19.17% |\n| 240ms | 10.80% | 5.91% | 8.00% | 13.84% | 15.17% |\n| **480ms** | **8.72%** | **4.90%** | **6.42%** | **10.45%** | **9.59%** |\n| 960ms | 7.70% | 4.34% | 5.68% | 8.99% | 6.80% |\n| 2400ms | 6.73% | 4.05% | 5.23% | 8.48% | 5.50% |\n\nAt **480ms** Voxtral Mini Realtime matches Mistral's offline Voxtral\nTranscribe 2.0 on Long-form English and Short-form English benchmarks\n(within 1 WER point on TEDLIUM, Meanwhile, AMI IHM, etc.).\n\n## Troubleshooting\n\n- **`Route: /v1/realtime` not registered** — your vLLM is < 0.16.0.\n Upgrade to 0.20.0+.\n- **Tokenizer initialization error** — you forgot `--tokenizer-mode mistral`.\n Voxtral Realtime's tokenizer can only load through `mistral_common`.\n- **`mistral_common` import error / wrong version** — install with the audio\n extras: `pip install -U \"mistral-common[audio]>=1.9.0\"`.\n- **Transformers v4 warning spam** — upgrade to Transformers v5\n (`uv pip install -U transformers`), tracked in\n [vllm-project/vllm#34642](https://github.com/vllm-project/vllm/issues/34642).\n- **Hangs / crashes on long sessions** — known upstream issue, see\n [#39996](https://github.com/vllm-project/vllm/issues/39996) (encoder KV\n cache eviction) and [#38233](https://github.com/vllm-project/vllm/issues/38233)\n (multi-session encode). Restart sessions periodically as a workaround.\n\n## References\n\n- [Mistral blog: Voxtral Transcribe 2](https://mistral.ai/news/voxtral-transcribe-2)\n- [vLLM Realtime API docs](https://docs.vllm.ai/en/latest/serving/openai_compatible_server/#realtime-api)\n- [vLLM streaming-input blog post](https://vllm.ai/blog/streaming-realtime.html)\n- [HuggingFace demo space](https://huggingface.co/spaces/mistralai/Voxtral-Mini-Realtime)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/mistralai/Voxtral-Mini-4B-Realtime-2602.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/mistralai/Voxtral-Mini-4B-Realtime-2602.yaml", "overlays": [] } }