{ "hf_id": "ByteDance-Seed/Seed-OSS-36B-Instruct", "meta": { "title": "Seed-OSS-36B", "slug": "seed-oss-36b", "provider": "Seed (ByteDance)", "description": "ByteDance Seed-OSS 36B dense model with unique 'thinking budget' control and 512K context support", "date_added": "2025-08-28", "date_updated": "2026-04-17", "tasks": [ "text" ], "related_recipes": [] }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve ByteDance-Seed/Seed-OSS-36B-Instruct \\\n --tensor-parallel-size 1 \\\n --enable-auto-tool-choice \\\n --tool-call-parser seed_oss", "argv": [ "vllm", "serve", "ByteDance-Seed/Seed-OSS-36B-Instruct", "--tensor-parallel-size", "1", "--enable-auto-tool-choice", "--tool-call-parser", "seed_oss" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest ByteDance-Seed/Seed-OSS-36B-Instruct \\\n --tensor-parallel-size 1 \\\n --enable-auto-tool-choice \\\n --tool-call-parser seed_oss", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "ByteDance-Seed/Seed-OSS-36B-Instruct", "--tensor-parallel-size", "1", "--enable-auto-tool-choice", "--tool-call-parser", "seed_oss" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "multi_node_tp": "/ByteDance-Seed/Seed-OSS-36B-Instruct/strategies/multi_node_tp.json", "kv_store_centralized_mooncake": "/ByteDance-Seed/Seed-OSS-36B-Instruct/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/ByteDance-Seed/Seed-OSS-36B-Instruct/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/ByteDance-Seed/Seed-OSS-36B-Instruct/hw/h200.json", "h100": "/ByteDance-Seed/Seed-OSS-36B-Instruct/hw/h100.json", "b200": "/ByteDance-Seed/Seed-OSS-36B-Instruct/hw/b200.json", "gb200": "/ByteDance-Seed/Seed-OSS-36B-Instruct/hw/gb200.json", "b300": "/ByteDance-Seed/Seed-OSS-36B-Instruct/hw/b300.json", "gb300": "/ByteDance-Seed/Seed-OSS-36B-Instruct/hw/gb300.json", "mi300x": "/ByteDance-Seed/Seed-OSS-36B-Instruct/hw/mi300x.json", "mi325x": "/ByteDance-Seed/Seed-OSS-36B-Instruct/hw/mi325x.json", "mi350x": "/ByteDance-Seed/Seed-OSS-36B-Instruct/hw/mi350x.json", "mi355x": "/ByteDance-Seed/Seed-OSS-36B-Instruct/hw/mi355x.json" } }, "model": { "model_id": "ByteDance-Seed/Seed-OSS-36B-Instruct", "min_vllm_version": "0.11.0", "architecture": "dense", "parameter_count": "36B", "active_parameters": "36B", "context_length": 524288, "base_args": [], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" }, "extras": [ { "note": "Pinned transformers commit required for Seed-OSS tokenizer compatibility", "command": "uv pip install git+https://github.com/huggingface/transformers.git@56d68c6706ee052b445e1e476056ed92ac5eb383" } ] } }, "features": { "tool_calling": { "description": "Seed-OSS tool-call parser with automatic tool choice", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "seed_oss" ] } }, "opt_in_features": [], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 86, "description": "Native BF16 weights on 8x GPU (TP=8)", "extra_args": [ "--tensor-parallel-size", "8" ] } }, "kv_offload_support": { "offloading_cpu": "verified", "offloading_fs": "verified" }, "hardware_overrides": { "amd": { "extra_args": [], "extra_env": { "VLLM_ROCM_USE_AITER": "1" } } }, "guide": "## Overview\n\nSeed-OSS-36B is a dense language model from ByteDance Seed with a unique **thinking\nbudget** feature for controlled reasoning, and up to 512K context. Users can choose\ntensor-parallel (low latency) or data-parallel (high throughput).\n\n## Prerequisites\n\n- Hardware: 8x GPU recommended (TP=8); also runs on consumer hardware like RTX 3090\n- vLLM >= 0.11.0 (support may require main branch)\n- Latest transformers for compatibility\n\n### Install vLLM (NVIDIA)\n\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto\nuv pip install git+https://github.com/huggingface/transformers.git@56d68c6706ee052b445e1e476056ed92ac5eb383\n```\n\n### Install vLLM (AMD ROCm)\n\n```bash\nuv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/\n```\n\n## Launch commands\n\nNVIDIA:\n\n```bash\nvllm serve ByteDance-Seed/Seed-OSS-36B-Instruct \\\n --host localhost --port 8000 \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser seed_oss\n```\n\nAMD:\n\n```bash\nexport VLLM_ROCM_USE_AITER=1\nvllm serve ByteDance-Seed/Seed-OSS-36B-Instruct \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice --tool-call-parser seed_oss \\\n --trust-remote-code\n```\n\nTuning:\n- `--max-model-len=65536` works well; max is 512K.\n- `--max-num-batched-tokens=32768` for prompt-heavy; reduce to 8K–16K for latency.\n- `--gpu-memory-utilization=0.95` to maximize KV cache.\n\n## Thinking Budget\n\nControl the model's chain-of-thought length via `chat_template_kwargs`. Recommended\nvalues are multiples of 512 (512, 1K, 2K, 4K, 8K, 16K). Use `0` for direct answers.\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\nmodel = client.models.list().data[0].id\n\nresponse = client.chat.completions.create(\n model=model,\n messages=[\n {\"role\": \"system\", \"content\": \"You are a helpful assistant\"},\n {\"role\": \"user\", \"content\": \"Janet's ducks lay 16 eggs per day...\"},\n ],\n extra_body={\"chat_template_kwargs\": {\"thinking_budget\": 512}},\n)\nprint(response.choices[0].message.content)\n```\n\ncurl:\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"ByteDance-Seed/Seed-OSS-36B-Instruct\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Explain quantum computing\"}],\n \"chat_template_kwargs\": {\"thinking_budget\": 512}\n }'\n```\n\nThe model emits `` blocks with `` markers that\nreport token usage against the budget.\n\n## Benchmarking\n\n```bash\nvllm bench serve \\\n --backend vllm --model ByteDance-Seed/Seed-OSS-36B-Instruct \\\n --endpoint /v1/completions --host localhost --port 8000 \\\n --dataset-name random --random-input 800 --random-output 100 \\\n --request-rate 2 --num-prompt 100\n```\n\n## References\n\n- [Seed-OSS-36B-Instruct on Hugging Face](https://huggingface.co/ByteDance-Seed/Seed-OSS-36B-Instruct)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "389ffebe555705c85337025d5df108d0384d365e", "path": "recipes/models/ByteDance-Seed/Seed-OSS-36B-Instruct.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/ByteDance-Seed/Seed-OSS-36B-Instruct.yaml", "overlays": [] } }