{ "hf_id": "autotrust/JEV-27B-VL", "meta": { "title": "JEV-27B-VL", "slug": "jev-27b-vl", "provider": "AutoTrust", "description": "Dense 27B vision-language decision model on a Qwen3.8-27B backbone, with a rank-32 LoRA (jev-decision) for calibrated choices over text and images.", "date_added": "2026-10-28", "date_updated": "2026-10-28", "tasks": [ "multimodal", "text" ], "performance_headline": "Qwen3.8-27B VL backbone; System 1 decisions through the jev-decision LoRA", "related_recipes": [ "Qwen/Qwen3.8-27B" ] }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve autotrust/JEV-27B-VL \\\n --served-model-name autotrust/JEV-27B-VL \\\n --enable-lora \\\n --max-lora-rank 32 \\\n --lora-modules jev-decision=JEV-27B-VL/adapter_vllm \\\n --logprobs-mode processed_logprobs \\\n --max-model-len 32768 \\\n --enable-prefix-caching \\\n --mamba-cache-mode align \\\n --limit-mm-per-prompt '{\"image\": 8}' \\\n --max-num-seqs 8 \\\n --trust-request-chat-template \\\n --tensor-parallel-size 1", "argv": [ "vllm", "serve", "autotrust/JEV-27B-VL", "--served-model-name", "autotrust/JEV-27B-VL", "--enable-lora", "--max-lora-rank", "32", "--lora-modules", "jev-decision=JEV-27B-VL/adapter_vllm", "--logprobs-mode", "processed_logprobs", "--max-model-len", "32768", "--enable-prefix-caching", "--mamba-cache-mode", "align", "--limit-mm-per-prompt", "{\"image\": 8}", "--max-num-seqs", "8", "--trust-request-chat-template", "--tensor-parallel-size", "1" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest autotrust/JEV-27B-VL \\\n --served-model-name autotrust/JEV-27B-VL \\\n --enable-lora \\\n --max-lora-rank 32 \\\n --lora-modules jev-decision=JEV-27B-VL/adapter_vllm \\\n --logprobs-mode processed_logprobs \\\n --max-model-len 32768 \\\n --enable-prefix-caching \\\n --mamba-cache-mode align \\\n --limit-mm-per-prompt '{\"image\": 8}' \\\n --max-num-seqs 8 \\\n --trust-request-chat-template \\\n --tensor-parallel-size 1", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "autotrust/JEV-27B-VL", "--served-model-name", "autotrust/JEV-27B-VL", "--enable-lora", "--max-lora-rank", "32", "--lora-modules", "jev-decision=JEV-27B-VL/adapter_vllm", "--logprobs-mode", "processed_logprobs", "--max-model-len", "32768", "--enable-prefix-caching", "--mamba-cache-mode", "align", "--limit-mm-per-prompt", "{\"image\": 8}", "--max-num-seqs", "8", "--trust-request-chat-template", "--tensor-parallel-size", "1" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "multi_node_tp": "/autotrust/JEV-27B-VL/strategies/multi_node_tp.json", "multi_node_tp_pp": "/autotrust/JEV-27B-VL/strategies/multi_node_tp_pp.json", "kv_store_centralized_mooncake": "/autotrust/JEV-27B-VL/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/autotrust/JEV-27B-VL/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/autotrust/JEV-27B-VL/hw/h200.json", "h100": "/autotrust/JEV-27B-VL/hw/h100.json", "b200": "/autotrust/JEV-27B-VL/hw/b200.json", "gb200": "/autotrust/JEV-27B-VL/hw/gb200.json", "b300": "/autotrust/JEV-27B-VL/hw/b300.json", "gb300": "/autotrust/JEV-27B-VL/hw/gb300.json", "mi300x": "/autotrust/JEV-27B-VL/hw/mi300x.json", "mi325x": "/autotrust/JEV-27B-VL/hw/mi325x.json", "mi350x": "/autotrust/JEV-27B-VL/hw/mi350x.json", "mi355x": "/autotrust/JEV-27B-VL/hw/mi355x.json" } }, "model": { "model_id": "autotrust/JEV-27B-VL", "min_vllm_version": "0.32.0", "architecture": "dense", "parameter_count": "27B", "active_parameters": "27B", "context_length": 262144, "base_args": [ "--served-model-name", "autotrust/JEV-27B-VL", "--enable-lora", "--max-lora-rank", "32", "--lora-modules", "jev-decision=JEV-27B-VL/adapter_vllm", "--logprobs-mode", "processed_logprobs", "--max-model-len", "32768", "--enable-prefix-caching", "--mamba-cache-mode", "align", "--limit-mm-per-prompt", "{\"image\": 8}", "--max-num-seqs", "8", "--trust-request-chat-template" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" }, "extras": [ { "note": "Local layout for LoRA module jev-decision. After this download the adapter is JEV-27B-VL/adapter_vllm; a ModelScope checkout uses the same adapter_vllm directory under whatever folder you chose.", "command": "hf download autotrust/JEV-27B-VL --local-dir JEV-27B-VL", "install_modes": [ "pip", "docker" ] } ] } }, "features": { "text_only": { "description": "Skip the vision encoder for text-only workloads and free its memory for KV cache.", "args": [ "--language-model-only" ] }, "encoder_parallel": { "description": "Run the vision encoder data-parallel across the tensor-parallel ranks instead of sharding it.", "args": [ "--mm-encoder-tp-mode", "data" ] } }, "opt_in_features": [ "text_only", "encoder_parallel" ], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 67, "description": "BF16 weights (about 52–56 GB). The model card's quick start targets one GPU with 80 GB or more; this recipe does not mark any GPU verified." } }, "guide": "## Overview\n\n[JEV-27B-VL](https://huggingface.co/autotrust/JEV-27B-VL) is AutoTrust's\nvision-language decision model. The checkpoint is a dense BF16\n`Qwen3_5ForConditionalGeneration` model (about 27.8B parameters, vision\ntower included) continued from\n[Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B). System 2 is\nthe served checkpoint, named `autotrust/JEV-27B-VL`. System 1 is the\nrank-32 LoRA `jev-decision` (`adapter_vllm/`, `r` 32, `lora_alpha` 64),\nused for calibrated choice probabilities over text or images.\n\nThis recipe has no `verified` hardware entry. The model card's quick start\nasks for one GPU with 80 GB or more, and the card's published latency\nfigures are from a B200. Those runs are the author's, not a badge on this\npage.\n\n## Prerequisites\n\n- **vLLM:** >= 0.32.0. Earlier releases are not supported for this\n checkpoint.\n- **Weights:** `hf download autotrust/JEV-27B-VL --local-dir JEV-27B-VL`\n so the LoRA path below exists. The same `adapter_vllm` directory is in\n the [ModelScope](https://modelscope.cn/models/autotrust/JEV-27B-VL)\n checkout; if that folder is not named `JEV-27B-VL`, point the module at\n `/adapter_vllm` (the HF snapshot path ends in\n `snapshots//adapter_vllm`).\n- **Docker:** the image only bind-mounts the Hugging Face cache. Mount the\n checkout into the container and keep `jev-decision` pointed at that\n `adapter_vllm` directory.\n- **Context:** `text_config.max_position_embeddings` is 262144. The\n validated command uses `--max-model-len 32768` (the `serve.sh` default).\n The card says a full 256K prompt needs about 17 GB of KV cache on top of\n the weights, and suggests `--max-model-len 131072` on an 80 GB GPU.\n\n## Decision LoRA and serve flags\n\nThese flags are the model card's validated `vllm serve` line. They are\nbase arguments, not optional toggles.\n\n- **`jev-decision`** — `--enable-lora`, `--max-lora-rank 32`, and\n `--lora-modules jev-decision=JEV-27B-VL/adapter_vllm`. The module name\n stays `jev-decision`. The path is the local checkout\n `JEV-27B-VL/adapter_vllm`, or the `adapter_vllm` directory inside an\n HF or ModelScope checkout.\n- **`--mamba-cache-mode align`** — the Qwen3.8 hybrid backbone mixes\n full attention with Mamba-style layers; the card serves it in `align`\n mode.\n- **`--limit-mm-per-prompt '{\"image\": 8}'`** — at most 8 images per\n request.\n- **`--max-num-seqs 8`** — required. The card says that with more than 8\n sequences in one batch, vLLM's LoRA path for this multimodal class\n returns wrong System 1 probabilities. Extra requests queue.\n- **`--logprobs-mode processed_logprobs`** — System 1 reads post-processor\n logprobs. On `/v1/chat/completions` or `/v1/completions` for\n `jev-decision`, the card also says to send `top_k: 0` and `top_p: 1.0`:\n `generation_config.json` sets `top_k` 20 and `top_p` 0.95, and those\n defaults would truncate the returned distribution.\n- **`--trust-request-chat-template`** — image decisions send a raw\n decision chat template on the request. System 2 keeps the checkpoint's\n own template.\n- **`--served-model-name autotrust/JEV-27B-VL`** — public name for\n System 2. `--enable-prefix-caching` matches the card's serve line.\n\n`POST /v1/decide` is not part of stock `vllm serve`. The checkpoint ships\n`serve_decide.py`, which starts the OpenAI server with these same flags\nand adds that route. Stock `vllm serve` still exposes System 2 as\n`autotrust/JEV-27B-VL` and System 1 as the LoRA model `jev-decision`.\n\n## Client usage\n\nSystem 2 (the base checkpoint), with an image:\n\n```python\nclient.chat.completions.create(\n model=\"autotrust/JEV-27B-VL\",\n messages=[{\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image_url\", \"image_url\": {\"url\": image_url}},\n {\"type\": \"text\", \"text\": \"What does this image show?\"},\n ],\n }],\n)\n```\n\nSystem 1 through the checkpoint's `/v1/decide` route (`serve_decide.py`):\n\n```bash\ncurl localhost:8000/v1/decide -H 'Content-Type: application/json' -d '{\n \"kind\": \"choice\",\n \"state\": \"Customer: my card was charged twice for one coffee.\",\n \"question\": \"Which team should handle this?\",\n \"options\": [\"billing\", \"shipping\", \"tech support\"]}'\n```\n\nText-only System 1 calls the LoRA directly (`model` `jev-decision`). The\ncard says those text decisions match\n[autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B).\n\n## References\n\n- [Hugging Face model card](https://huggingface.co/autotrust/JEV-27B-VL)\n- [ModelScope (autotrust/JEV-27B-VL)](https://modelscope.cn/models/autotrust/JEV-27B-VL)\n- [ModelScope mirror listed on the card (AutoTrust-Labs/JEV-27B-VL)](https://www.modelscope.ai/models/AutoTrust-Labs/JEV-27B-VL)\n- [Base model: Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/autotrust/JEV-27B-VL.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/autotrust/JEV-27B-VL.yaml", "overlays": [] } }