# Google/translategemma-27b-it — vLLM serving recipe

> Lightweight open translation model from Google (based on Gemma 3) supporting 55 languages. Served via the vLLM-optimized Infomaniak-AI checkpoint.

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16)

- Interactive page: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/Google/translategemma-27b-it
- This configuration as JSON: https://vllm-recipes-cimvhk5sf-inferact-inc.vercel.app/Google/translategemma-27b-it.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `vllm_optimized`, `small_4b`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)

## Install

Requires vLLM >= 0.14.1.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve google/translategemma-27b-it \
  --tensor-parallel-size 1
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest google/translategemma-27b-it \
  --tensor-parallel-size 1
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[TranslateGemma](https://huggingface.co/collections/google/translategemma) is a family of lightweight, state-of-the-art open translation models from Google, based on the Gemma 3 family. TranslateGemma models handle translation across 55 languages and are small enough to deploy on laptops, desktops, and modest cloud GPU environments.

#### Original Models
- [google/translategemma-27b-it](https://huggingface.co/google/translategemma-27b-it)
- [google/translategemma-4b-it](https://huggingface.co/google/translategemma-4b-it)

#### Optimized vLLM Models
- [Infomaniak-AI/vllm-translategemma-27b-it](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it)
- [Infomaniak-AI/vllm-translategemma-4b-it](https://huggingface.co/Infomaniak-AI/vllm-translategemma-4b-it)

#### Why use the vLLM-optimized models?

The original Google models have compatibility issues with standard inference engines like vLLM. The optimized versions from Infomaniak-AI ([detailed changes](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it#changes-from-original-model)):

- **vLLM Compatibility**: Originals require custom JSON parameters (`source_lang_code`/`target_lang_code`). The optimized version uses string delimiters.
- **RoPE Simplification**: Originals use a complex RoPE configuration for sliding attention. Optimized uses a standard linear RoPE format (`factor: 8.0`).
- **EOS Token Fix**: Corrects the EOS token from `<end_of_turn>` to `<eos>`.

### Prerequisites

#### Docker
```bash
docker pull vllm/vllm-openai:latest
```

### Deployment Configurations

Verified for both 4B and 27B:
```bash
docker run -itd --name google-translategemma-27b-it \
  --ipc=host \
  --network host \
  --shm-size 16G \
  --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
    Infomaniak-AI/vllm-translategemma-27b-it \
    --served-model-name translategemma-27b-it \
    --gpu-memory-utilization 0.8 \
    --host 0.0.0.0 \
    --port 8000
```

### Client Usage

Tips:
- **Prompt Delimiters**: Encode language metadata directly in the content string:
  `<<<source>>>{src_lang}<<<target>>>{tgt_lang}<<<text>>>{text}`
- **Language Codes**: ISO 639-1 Alpha-2 (e.g. `en`, `zh`) and regional variants (e.g. `en_US`, `zh_CN`).
- **Context Limit**: ~2K tokens.

#### cURL
```bash
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "translategemma-27b-it",
    "messages": [{
      "role": "user",
      "content": "<<<source>>>en<<<target>>>zh<<<text>>>We distribute two models for language identification, which can recognize 176 languages."
    }]
  }'
```

### References

- [TranslateGemma collection](https://huggingface.co/collections/google/translategemma)
- [Infomaniak-AI optimized 27B](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it)
- [Infomaniak-AI optimized 4B](https://huggingface.co/Infomaniak-AI/vllm-translategemma-4b-it)