{ "hf_id": "tencent/Hunyuan-A13B-Instruct", "meta": { "title": "Hunyuan-A13B-Instruct", "slug": "hunyuan-a13b-instruct", "provider": "Hunyuan (Tencent)", "description": "Tencent Hunyuan A13B instruct-tuned MoE language model with AITER-accelerated AMD ROCm deployment", "date_added": "2026-04-11", "date_updated": "2026-09-10", "tasks": [ "text" ], "performance_headline": "Hunyuan-A13B MoE with AITER acceleration on AMD MI300X/MI325X/MI355X", "related_recipes": [], "hardware": { "mi300x": "verified", "mi325x": "verified", "mi355x": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve tencent/Hunyuan-A13B-Instruct \\\n --trust-remote-code \\\n --tensor-parallel-size 8", "argv": [ "vllm", "serve", "tencent/Hunyuan-A13B-Instruct", "--trust-remote-code", "--tensor-parallel-size", "8" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest tencent/Hunyuan-A13B-Instruct \\\n --trust-remote-code \\\n --tensor-parallel-size 8", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "tencent/Hunyuan-A13B-Instruct", "--trust-remote-code", "--tensor-parallel-size", "8" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tep": "/tencent/Hunyuan-A13B-Instruct/strategies/single_node_tep.json", "multi_node_tp": "/tencent/Hunyuan-A13B-Instruct/strategies/multi_node_tp.json", "multi_node_tep": "/tencent/Hunyuan-A13B-Instruct/strategies/multi_node_tep.json", "multi_node_dep": "/tencent/Hunyuan-A13B-Instruct/strategies/multi_node_dep.json", "kv_store_centralized_mooncake": "/tencent/Hunyuan-A13B-Instruct/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/tencent/Hunyuan-A13B-Instruct/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/tencent/Hunyuan-A13B-Instruct/hw/h200.json", "h100": "/tencent/Hunyuan-A13B-Instruct/hw/h100.json", "b200": "/tencent/Hunyuan-A13B-Instruct/hw/b200.json", "gb200": "/tencent/Hunyuan-A13B-Instruct/hw/gb200.json", "b300": "/tencent/Hunyuan-A13B-Instruct/hw/b300.json", "gb300": "/tencent/Hunyuan-A13B-Instruct/hw/gb300.json", "mi300x": "/tencent/Hunyuan-A13B-Instruct/hw/mi300x.json", "mi325x": "/tencent/Hunyuan-A13B-Instruct/hw/mi325x.json", "mi350x": "/tencent/Hunyuan-A13B-Instruct/hw/mi350x.json", "mi355x": "/tencent/Hunyuan-A13B-Instruct/hw/mi355x.json" } }, "model": { "model_id": "tencent/Hunyuan-A13B-Instruct", "min_vllm_version": "0.11.0", "architecture": "moe", "parameter_count": "80B", "active_parameters": "13B", "context_length": 32768, "base_args": [ "--trust-remote-code" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "long_context": { "description": "Raise the context window from the config's default 32,768 to the model's full 262,144 tokens. The 32K cap in config.json exists to avoid OOM on smaller nodes, so enable this only with enough KV-cache headroom.", "args": [ "--max-model-len", "262144" ], "env": { "VLLM_ALLOW_LONG_MAX_MODEL_LEN": "1" } } }, "opt_in_features": [ "long_context" ], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 580, "description": "Full precision BF16 — 2x GPU (TP=2) on AMD MI300X/MI325X/MI355X" }, "fp8": { "model_id": "tencent/Hunyuan-A13B-Instruct-FP8", "precision": "fp8", "vram_minimum_gb": 96, "description": "FP8 quantized weights for Hopper/Blackwell GPUs", "extra_args": [ "--kv-cache-dtype", "fp8" ], "json": "/tencent/Hunyuan-A13B-Instruct-FP8.json" } }, "kv_offload_support": { "offloading_cpu": "verified", "offloading_fs": "verified" }, "hardware_overrides": { "amd": { "extra_args": [], "extra_env": { "VLLM_ROCM_USE_AITER": "1" } } }, "guide": "## Overview\n\nHunyuan-A13B-Instruct is Tencent's instruct-tuned Hunyuan MoE model. This recipe\ncovers deployment on AMD ROCm GPUs (MI300X / MI325X / MI355X) with\n[AITER](https://github.com/ROCm/aiter) acceleration enabled via\n`VLLM_ROCM_USE_AITER=1`.\n\n## Prerequisites\n\n- **vLLM version:** ROCm build\n- **Python:** 3.12\n- **Hardware:** AMD MI300X / MI325X / MI355X\n- **ROCm:** 7.0+, glibc >= 2.35 (or use Docker)\n\n### Install vLLM (ROCm)\n\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/\n```\n\nIf the environment does not meet the Python/ROCm/glibc requirements, use the\nDocker-based setup from the vLLM install docs.\n\n## Launching the Server\n\n```bash\nexport VLLM_ROCM_USE_AITER=1\nvllm serve tencent/Hunyuan-A13B-Instruct \\\n --tensor-parallel-size 2 \\\n --trust-remote-code\n```\n\n## Benchmarking\n\n```bash\nvllm bench serve \\\n --model \"tencent/Hunyuan-A13B-Instruct\" \\\n --dataset-name random \\\n --random-input-len 8000 \\\n --random-output-len 1000 \\\n --request-rate 10000 \\\n --num-prompts 16 \\\n --ignore-eos\n```\n\n## Troubleshooting\n\n- **First launch delay:** AITER JIT-compiles optimized kernels on first launch, which\n can take several minutes. Subsequent runs use cached kernels.\n- **Environment mismatch:** If wheel install fails, fall back to the vLLM ROCm Docker image.\n\n## References\n\n- [Model card](https://huggingface.co/tencent/Hunyuan-A13B-Instruct)\n- [AITER](https://github.com/ROCm/aiter)\n- [vLLM install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/tencent/Hunyuan-A13B-Instruct.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/tencent/Hunyuan-A13B-Instruct.yaml", "overlays": [] } }