# moonshotai/Kimi-Linear-48B-A3B-Instruct — vLLM serving recipe

> Kimi-Linear is a 48B-parameter instruction-tuned MoE model (~3B activated) with a linear-attention variant supporting very long context (1M tokens).

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16)

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/moonshotai/Kimi-Linear-48B-A3B-Instruct
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/moonshotai/Kimi-Linear-48B-A3B-Instruct.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `strategy`: `single_node_tp`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_tep`, `multi_node_dep` (multi-node strategies also need `nodes=<n>`)

## Install

Requires vLLM >= 0.11.2.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

Extra dependencies:

- Pin vllm==0.11.2 — 0.12.0 has a known Kimi-Linear regression

```bash
uv pip install vllm==0.11.2 --torch-backend auto
```

## Serve

```bash
vllm serve moonshotai/Kimi-Linear-48B-A3B-Instruct \
  --trust-remote-code \
  --tensor-parallel-size 1
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest moonshotai/Kimi-Linear-48B-A3B-Instruct \
  --trust-remote-code \
  --tensor-parallel-size 1
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Kimi-Linear is Moonshot AI's 48B-parameter instruction-tuned MoE model (`A3B` indicates
~3B active parameters per token) featuring a linear-attention variant that enables very
long context windows (up to 1,048,576 tokens).

### Prerequisites

- **Hardware**: 4 or 8 GPUs on a single node
- **vLLM**: `0.11.2` recommended. Avoid vLLM `0.12.0`, which has a known bug
  `MLAModules.__init__() missing 1 required positional argument: 'indexer_rotary_emb'`
  that affects Kimi-Linear.

```bash
uv venv
source .venv/bin/activate
# Install a stable version (avoid 0.12.0)
uv pip install vllm==0.11.2 --torch-backend auto
```

#### AMD ROCm

The ROCm wheel requires Python 3.12, ROCm 7.0+, and glibc >= 2.35. Use the
ROCm Docker image when your host environment does not meet those requirements.

```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm==0.11.2 --extra-index-url https://wheels.vllm.ai/rocm/
```

### Running Kimi-Linear

The following snippets assume 4 or 8 GPUs on a single node.

#### 4-GPU Tensor Parallel

```bash
vllm serve moonshotai/Kimi-Linear-48B-A3B-Instruct \
  --port 8000 \
  --tensor-parallel-size 4 \
  --max-model-len 1048576 \
  --trust-remote-code
```

#### 8-GPU Tensor Parallel

```bash
vllm serve moonshotai/Kimi-Linear-48B-A3B-Instruct \
  --port 8000 \
  --tensor-parallel-size 8 \
  --max-model-len 1048576 \
  --trust-remote-code
```

#### ROCm on 8x MI300X / MI325X / MI355X

```bash
export SAFETENSORS_FAST_GPU=1
vllm serve moonshotai/Kimi-Linear-48B-A3B-Instruct \
  --tensor-parallel-size 8 \
  --max-model-len auto \
  --trust-remote-code
```

If you see OOM, reduce `--max-model-len` (e.g. 65536) or increase
`--gpu-memory-utilization` (<= 0.95).

### Client Usage

Once the server is up, test it with:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"moonshotai/Kimi-Linear-48B-A3B-Instruct","messages":[{"role":"user","content":"Hello!"}]}'
```

### Troubleshooting

- **`MLAModules.__init__() missing 1 required positional argument: 'indexer_rotary_emb'`**:
  Known bug in vLLM 0.12.0 affecting Kimi-Linear. Pin to `vllm==0.11.2` instead.
- **OOM**: Reduce `--max-model-len` or increase `--gpu-memory-utilization` up to 0.95.

### References

- [Kimi-Linear-48B-A3B-Instruct on Hugging Face](https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct)