# tencent/Hy3-preview — vLLM serving recipe

> Tencent Hunyuan Hy3-preview — scaled-up MoE language model (295B total / 21B active) with a 3.8B MTP layer for speculative decoding, 256K context, and hy_v3 tool/reasoning parsers

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning` · frontend `rust`

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/tencent/Hy3-preview
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/tencent/Hy3-preview.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tep`, `multi_node_dep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.20.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve tencent/Hy3-preview \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser hy_v3 \
  --enable-auto-tool-choice \
  --reasoning-parser hy_v3
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:latest tencent/Hy3-preview \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser hy_v3 \
  --enable-auto-tool-choice \
  --reasoning-parser hy_v3
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

## Hy3-preview Usage Guide

Hy3-preview is Tencent Hunyuan's latest open-source Mixture-of-Experts language model:
295B total parameters with 21B activated per token, plus a 3.8B MTP layer for
speculative decoding. 80 transformer layers, 192 routed experts (top-8) + 1 shared
expert, GQA with 64 heads over 8 KV heads, 256K context.

A pretrained base checkpoint is published at
[tencent/Hy3-preview-Base](https://huggingface.co/tencent/Hy3-preview-Base); this
recipe covers the instruct model.

### Setup

Choose one of the following setup methods.

#### Using Docker

```bash
docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:hy3-preview tencent/Hy3-preview \
    --tensor-parallel-size 8 \
    --tool-call-parser hy_v3 \
    --reasoning-parser hy_v3 \
    --enable-auto-tool-choice \
    --served-model-name hy3-preview
```

#### Installing from source

```bash
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
git clone https://github.com/vllm-project/vllm.git
cd vllm
uv pip install --editable . --torch-backend=auto
```

### Model Deployment

To serve Hy3-preview on 8 GPUs, use H20-3e(141GB), H200, AMD MI300X/MI325X (192 GB),
AMD MI350X/MI355X (288 GB), or other GPUs with larger memory capacity. Smaller-memory
8-GPU configurations (8×H100 80GB, 8×A100 80GB) do not fit the BF16 weights plus KV
cache — use multi-node TP for those.

#### Serving on 8×AMD MI300X / MI325X / MI350X / MI355X

Hy3-preview support is being added in vLLM PR
[#40681](https://github.com/vllm-project/vllm/pull/40681). Until it merges, AMD users
must build vLLM from the PR branch inside the published ROCm vLLM nightly image
(`rocm/vllm-dev:nightly`). See the AMD install note in `hardware_overrides.amd`
above for the full reproducer.

Once vLLM is installed, serve with the standard launcher plus the AITER environment
variables (the recipe's `hardware_overrides.amd.extra_env` applies these
automatically when the AMD profile is selected on the recipe site):

```bash
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=1
export VLLM_ROCM_USE_AITER_MHA=1
export VLLM_ROCM_USE_AITER_RMSNORM=1
export VLLM_ROCM_USE_AITER_LINEAR=1

vllm serve tencent/Hy3-preview \
  --tensor-parallel-size 8 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --served-model-name hy3-preview \
  --gpu-memory-utilization 0.90
```

MTP (recommended on AMD for lower latency, same flags as the NVIDIA path):

```bash
vllm serve tencent/Hy3-preview \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 1 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --served-model-name hy3-preview \
  --gpu-memory-utilization 0.90
```

#### Serving on 8×H200 or 8×H20-3e(141GB)

Without Multi-Token Prediction (MTP):

```bash
vllm serve tencent/Hy3-preview \
  --tensor-parallel-size 8 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --served-model-name hy3-preview
```

With MTP (recommended for lower latency):

```bash
vllm serve tencent/Hy3-preview \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 1 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --served-model-name hy3-preview
```

### Sampling and Reasoning Modes

Tencent's recommended sampling parameters: `temperature=0.9`, `top_p=1.0`.

Reasoning is controlled via `chat_template_kwargs.reasoning_effort`:

| Value | Behavior |
|:---|:---|
| `no_think` (default) | Direct response, no chain-of-thought |
| `low` | Light reasoning |
| `high` | Deep chain-of-thought for math/coding/complex reasoning |

When tools are registered, set `interleaved_thinking: true` to allow the model to
think between tool calls.

#### OpenAI Client Example

```bash
uv pip install -U openai
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello."},
]

# Direct response (default).
resp = client.chat.completions.create(
    model="hy3-preview",
    messages=messages,
    temperature=0.9,
    top_p=1.0,
    max_tokens=4096,
)
print(resp.choices[0].message.content)

# Deep reasoning: set reasoning_effort (and interleaved_thinking if using tools).
resp_think = client.chat.completions.create(
    model="hy3-preview",
    messages=messages,
    temperature=0.9,
    top_p=1.0,
    max_tokens=4096,
    extra_body={
      "chat_template_kwargs": {
          "reasoning_effort": "high",
          "interleaved_thinking": True,
      },
    },
)
output_msg = resp_think.choices[0].message
print(output_msg.reasoning)           # chain-of-thought
print(output_msg.content)             # final answer
```

#### cURL Usage

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hy3-preview",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello."}
    ],
    "temperature": 0.9,
    "top_p": 1.0,
    "max_tokens": 4096
  }'
```

### Benchmarking

For benchmarking, disable prefix caching by adding `--no-enable-prefix-caching` to
the server command.

The following uses 8×H20-3e(141GB) as an example.

```bash
vllm bench serve \
  --model tencent/Hy3-preview \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 1024 \
  --max-concurrency 32 \
  --num-prompts 160 \
  --served-model-name hy3-preview
```

Representative output:

```shell
============ Serving Benchmark Result ============
Successful requests:                     160       
Failed requests:                         0         
Maximum request concurrency:             32        
Benchmark duration (s):                  280.58    
Total input tokens:                      1310720   
Total generated tokens:                  163840    
Request throughput (req/s):              0.57      
Output token throughput (tok/s):         583.93    
Peak output token throughput (tok/s):    1024.00   
Peak concurrent requests:                36.00     
Total token throughput (tok/s):          5255.36   
---------------Time to First Token----------------
Mean TTFT (ms):                          4542.41   
Median TTFT (ms):                        2762.17   
P99 TTFT (ms):                           21062.96  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          50.34     
Median TPOT (ms):                        51.77     
P99 TPOT (ms):                           54.07     
---------------Inter-token Latency----------------
Mean ITL (ms):                           50.34     
Median ITL (ms):                         34.32     
P99 ITL (ms):                            689.10    
==================================================
```
### References

- [Hugging Face model card](https://huggingface.co/tencent/Hy3-preview)
- [Hy3-preview-Base (pretrained)](https://huggingface.co/tencent/Hy3-preview-Base)
- [GitHub: Tencent-Hunyuan/Hy3-preview](https://github.com/Tencent-Hunyuan/Hy3-preview)
- [Tencent Hunyuan](https://hy.tencent.com/research/hy3)