{ "hf_id": "JetBrains/Mellum2-12B-A2.5B-Thinking", "meta": { "title": "Mellum2-12B-A2.5B-Thinking", "slug": "mellum2-12b-a2.5b-thinking", "provider": "JetBrains", "description": "JetBrains' reasoning-augmented code MoE (12B total / 2.5B active) that emits explicit chains for debugging, planning, and agentic coding", "date_added": "2026-06-02", "date_updated": "2026-06-02", "tasks": [ "text" ], "performance_headline": "69.9 LiveCodeBench v6, 58.4 AIME — fits on a single GPU", "related_recipes": [ "JetBrains/Mellum2-12B-A2.5B-Instruct" ], "hardware": { "h200": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve JetBrains/Mellum2-12B-A2.5B-Thinking \\\n --max-model-len 131072 \\\n --tensor-parallel-size 1 \\\n --reasoning-parser qwen3 \\\n --enable-auto-tool-choice \\\n --tool-call-parser hermes", "argv": [ "vllm", "serve", "JetBrains/Mellum2-12B-A2.5B-Thinking", "--max-model-len", "131072", "--tensor-parallel-size", "1", "--reasoning-parser", "qwen3", "--enable-auto-tool-choice", "--tool-call-parser", "hermes" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest JetBrains/Mellum2-12B-A2.5B-Thinking \\\n --max-model-len 131072 \\\n --tensor-parallel-size 1 \\\n --reasoning-parser qwen3 \\\n --enable-auto-tool-choice \\\n --tool-call-parser hermes", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "JetBrains/Mellum2-12B-A2.5B-Thinking", "--max-model-len", "131072", "--tensor-parallel-size", "1", "--reasoning-parser", "qwen3", "--enable-auto-tool-choice", "--tool-call-parser", "hermes" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tep": "/JetBrains/Mellum2-12B-A2.5B-Thinking/strategies/single_node_tep.json", "single_node_dep": "/JetBrains/Mellum2-12B-A2.5B-Thinking/strategies/single_node_dep.json", "kv_store_centralized_mooncake": "/JetBrains/Mellum2-12B-A2.5B-Thinking/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/JetBrains/Mellum2-12B-A2.5B-Thinking/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/JetBrains/Mellum2-12B-A2.5B-Thinking/hw/h200.json", "h100": "/JetBrains/Mellum2-12B-A2.5B-Thinking/hw/h100.json", "b200": "/JetBrains/Mellum2-12B-A2.5B-Thinking/hw/b200.json", "gb200": "/JetBrains/Mellum2-12B-A2.5B-Thinking/hw/gb200.json", "b300": "/JetBrains/Mellum2-12B-A2.5B-Thinking/hw/b300.json", "gb300": "/JetBrains/Mellum2-12B-A2.5B-Thinking/hw/gb300.json", "mi300x": "/JetBrains/Mellum2-12B-A2.5B-Thinking/hw/mi300x.json", "mi325x": "/JetBrains/Mellum2-12B-A2.5B-Thinking/hw/mi325x.json", "mi350x": "/JetBrains/Mellum2-12B-A2.5B-Thinking/hw/mi350x.json", "mi355x": "/JetBrains/Mellum2-12B-A2.5B-Thinking/hw/mi355x.json" } }, "model": { "model_id": "JetBrains/Mellum2-12B-A2.5B-Thinking", "min_vllm_version": "0.23.0", "architecture": "moe", "parameter_count": "12B", "active_parameters": "2.5B", "context_length": 131072, "base_args": [ "--max-model-len", "131072" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "reasoning": { "description": "Parse the model's ... reasoning blocks (Qwen3-style parser)", "args": [ "--reasoning-parser", "qwen3" ] }, "tool_calling": { "description": "Hermes tool-call parser for function calling", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "hermes" ] }, "spec_decoding": { "description": "DFlash speculative decoding with an externally trained draft model; the served checkpoint is unchanged.", "args": [ "--speculative-config", "{\"model\": \"RedHatAI/Mellum2-12B-A2.5B-Thinking-speculator.dflash\", \"num_speculative_tokens\": 7, \"method\": \"dflash\"}" ] } }, "opt_in_features": [ "spec_decoding" ], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 29, "description": "Native bfloat16 weights; fits comfortably on a single H200/H100/A100" }, "fp8": { "model_id": "RedHatAI/Mellum2-12B-A2.5B-Thinking-FP8-Dynamic", "precision": "fp8", "vram_minimum_gb": 15, "description": "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization", "extra_args": [ "--kv-cache-dtype", "fp8" ], "json": "/RedHatAI/Mellum2-12B-A2.5B-Thinking-FP8-Dynamic.json" }, "nvfp4": { "model_id": "RedHatAI/Mellum2-12B-A2.5B-Thinking-NVFP4", "precision": "nvfp4", "vram_minimum_gb": 8, "description": "RedHatAI NVFP4 quantization", "json": "/RedHatAI/Mellum2-12B-A2.5B-Thinking-NVFP4.json" } }, "guide": "## Overview\n\n[Mellum2-12B-A2.5B-Thinking](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking)\nis JetBrains' reasoning-augmented code assistant. It uses a Mixture-of-Experts architecture\nwith 64 experts (8 activated per token) — 12B total parameters, 2.5B active — combining\nsliding-window and full-attention layers for a 131,072-token context. The model emits its\nchain-of-thought inside `...` blocks before the final answer, making it\nsuited to complex debugging, multi-step planning, and agentic workflows. For direct,\nlow-latency answers without reasoning traces, use the\n[Instruct](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct) variant instead.\n\n## Prerequisites\n\n- Hardware: a single H200, H100, or A100 (~29 GB at bf16) is plenty\n- vLLM **nightly** — `MellumForCausalLM` support landed after v0.22.0 and is not yet in a\n stable release. Install the nightly wheels until the next tagged release ships.\n\n### Install vLLM (nightly)\n\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly\n```\n\n## Launch command\n\n```bash\n# With reasoning (recommended for the Thinking checkpoint)\nvllm serve JetBrains/Mellum2-12B-A2.5B-Thinking \\\n --max-model-len 131072 \\\n --reasoning-parser qwen3\n\n# Add tool calling\nvllm serve JetBrains/Mellum2-12B-A2.5B-Thinking \\\n --max-model-len 131072 \\\n --reasoning-parser qwen3 \\\n --enable-auto-tool-choice \\\n --tool-call-parser hermes\n```\n\n## Client usage\n\nJetBrains recommends sampling at `temperature=0.6`, `top_p=0.95`, `top_k=20` for the\nThinking checkpoint.\n\n```python\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"EMPTY\")\n\nresp = client.chat.completions.create(\n model=\"JetBrains/Mellum2-12B-A2.5B-Thinking\",\n messages=[{\"role\": \"user\", \"content\": \"Is 1024 a power of 2? Explain your reasoning.\"}],\n max_tokens=81920,\n temperature=0.6,\n top_p=0.95,\n extra_body={\"top_k\": 20},\n)\nprint(resp.choices[0].message.content)\n```\n\n## References\n\n- [Model card](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking)\n- [vLLM support PR #43992](https://github.com/vllm-project/vllm/pull/43992)\n- [Mellum2 Technical Report](https://arxiv.org/abs/2605.31268)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/JetBrains/Mellum2-12B-A2.5B-Thinking.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/JetBrains/Mellum2-12B-A2.5B-Thinking.yaml", "overlays": [] } }