# OpenGVLab/InternVL3_5-8B — vLLM serving recipe

> InternVL 3.5 vision-language models from Shanghai AI Lab with thinking-mode prompting

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16)

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/OpenGVLab/InternVL3_5-8B
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/OpenGVLab/InternVL3_5-8B.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `text_only`, `encoder_parallel` (default: none)

## Install

Requires vLLM >= 0.10.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve OpenGVLab/InternVL3_5-8B \
  --trust-remote-code \
  --tensor-parallel-size 1
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest OpenGVLab/InternVL3_5-8B \
  --trust-remote-code \
  --tensor-parallel-size 1
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[InternVL3.5](https://github.com/OpenGVLab/InternVL) is a vision-language model developed
by Shanghai AI Laboratory. It supports single-image and multi-image prompts, plus an
optional "thinking mode" via a custom system prompt.

### Prerequisites

- Hardware: 1x GPU with >=20 GB VRAM (A100, L40S, H100, etc.)
- vLLM >= 0.10.0

#### Install vLLM (CUDA)

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### Install vLLM (AMD ROCm MI300X/MI325X/MI355X)

```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700
```

### Launch command

```bash
vllm serve OpenGVLab/InternVL3_5-8B --trust-remote-code
```

On AMD:

```bash
export VLLM_ROCM_USE_AITER=1
vllm serve OpenGVLab/InternVL3_5-8B --trust-remote-code
```

### Client Usage

Single image:

```python
from openai import OpenAI
client = OpenAI(api_key="", base_url="http://0.0.0.0:8000/v1")
model_name = client.models.list().data[0].id

response = client.chat.completions.create(
    model=model_name,
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe the image."},
            {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg"}},
        ],
    }],
    temperature=0.0,
)
print(response.choices[0].message.content)
```

### Thinking Mode

Set a thinking system prompt and use `temperature=0.6` to mitigate repetition:

```python
THINKING_SYSTEM_PROMPT = """
You are an AI assistant that rigorously follows this response protocol:

1. First, conduct a detailed analysis of the question. Consider different angles, potential
solutions, and reason through the problem step-by-step. Enclose this entire thinking process
within <think> and </think> tags.

2. After the thinking section, provide a clear, concise, and direct answer to the user's
question. Separate the answer from the think section with a newline.
""".strip()
```

### References

- [InternVL3.5-8B](https://huggingface.co/OpenGVLab/InternVL3_5-8B)
- [InternVL GitHub](https://github.com/OpenGVLab/InternVL)