{
  "hf_id": "nvidia/Cosmos3-Nano",
  "meta": {
    "title": "Cosmos3-Nano",
    "slug": "cosmos3-nano",
    "provider": "NVIDIA",
    "description": "Compact 16B omnimodal world model (Mixture-of-Transformers) for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI",
    "date_added": "2026-06-02",
    "date_updated": "2026-06-02",
    "tasks": [
      "omni"
    ],
    "performance_headline": "16B omnimodal world model — single-GPU H200 video/audio generation",
    "related_recipes": [
      "nvidia/Cosmos3-Super",
      "nvidia/Cosmos3-Super-Text2Image",
      "nvidia/Cosmos3-Super-Image2Video"
    ],
    "hardware": {
      "h200": "verified",
      "h100": "verified"
    }
  },
  "model": {
    "model_id": "nvidia/Cosmos3-Nano",
    "min_vllm_version": "0.21.0",
    "docker_image": "vllm/vllm-omni:cosmos3",
    "architecture": "dense",
    "parameter_count": "16B",
    "active_parameters": "16B",
    "context_length": 262144,
    "base_args": [
      "--host",
      "0.0.0.0",
      "--port",
      "8000",
      "--init-timeout",
      "1800"
    ],
    "base_env": {},
    "install": {
      "pip": {
        "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto"
      },
      "docker": {
        "command": "docker pull vllm/vllm-omni:cosmos3",
        "note": "Release-tested vLLM-Omni image bundling the Cosmos3 omni handlers."
      }
    }
  },
  "omni": {
    "tasks": [
      {
        "id": "t2v",
        "description": "Text → video (audio muxed into the MP4)",
        "curl": "curl -X POST http://localhost:8000/v1/videos/sync \\\n  -H \"Accept: video/mp4\" \\\n  -F \"prompt=$(cat assets/example_t2v_prompt.json)\" \\\n  -F \"negative_prompt=$(cat assets/negative_prompt.json)\" \\\n  -F \"size=1280x720\" \\\n  -F \"num_frames=189\" \\\n  -F \"fps=24\" \\\n  -F \"num_inference_steps=35\" \\\n  -F \"guidance_scale=6.0\" \\\n  -F \"max_sequence_length=4096\" \\\n  -F \"flow_shift=10.0\" \\\n  -F \"seed=123\" \\\n  --output output.mp4\n"
      },
      {
        "id": "i2v",
        "description": "Image → video (temporally coherent, audio muxed in)",
        "curl": "curl -X POST http://localhost:8000/v1/videos/sync \\\n  -H \"Accept: video/mp4\" \\\n  -F \"input_reference=@assets/example_i2v_input.jpg\" \\\n  -F \"prompt=$(cat assets/example_i2v_prompt.json)\" \\\n  -F \"negative_prompt=$(cat assets/negative_prompt.json)\" \\\n  -F \"size=1280x720\" \\\n  -F \"num_frames=189\" \\\n  -F \"fps=24\" \\\n  -F \"num_inference_steps=35\" \\\n  -F \"guidance_scale=6.0\" \\\n  -F \"max_sequence_length=4096\" \\\n  -F \"flow_shift=10.0\" \\\n  -F \"seed=1111\" \\\n  --output output.mp4\n"
      }
    ]
  },
  "features": {},
  "opt_in_features": [],
  "variants": {
    "default": {
      "precision": "bf16",
      "vram_minimum_gb": 39,
      "description": "BF16 weights — single-GPU H200 omni generation"
    }
  },
  "hardware_overrides": {},
  "guide": "## Overview\n\n**Cosmos3-Nano** is the compact (16B) member of NVIDIA's Cosmos3 family of\n**omnimodal world models**. Built on a Mixture-of-Transformers (MoT)\narchitecture — an autoregressive tower for discrete-token generation paired\nwith a diffusion tower for continuous modalities — it generates coherent text,\nimages, video, audio, and action commands from combinations of text, image,\nvideo, and action-trajectory inputs. It targets Physical AI: world\nunderstanding, world simulation, future prediction, and action reasoning.\n\nThis recipe covers the **vLLM-Omni** serving path (video + muxed-audio\ngeneration). The same checkpoint also serves a text/multimodal **Reasoner**\nendpoint via standard vLLM — see [Reasoner mode](#reasoner-mode-multimodal-understanding)\nbelow.\n\n## Prerequisites\n\n- 1× H200 (or H100 / A100) for the documented single-GPU profile\n- The release-tested `vllm/vllm-omni:cosmos3` container, or vLLM-Omni installed\n  on top of `vllm==0.21.0`\n- Prompts should be JSON-upsampled for best quality (see\n  [cosmos-framework prompt upsampling](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/prompt_upsampling.md))\n\n## Launch command (vLLM-Omni)\n\n```bash\ndocker pull vllm/vllm-omni:cosmos3\n\nvllm serve nvidia/Cosmos3-Nano \\\n  --omni \\\n  --host 0.0.0.0 \\\n  --port 8000 \\\n  --init-timeout 1800\n```\n\nTo speed up inference with additional GPUs, enable context parallelism with\n`--ulysses-degree` or switch to tensor parallelism with\n`--tensor-parallel-size`. On memory-constrained GPUs, `--enable-layerwise-offload`\nreduces VRAM usage at a performance cost.\n\n### Example: text-to-video\n\n```bash\ncurl -X POST http://localhost:8000/v1/videos/sync \\\n  -H \"Accept: video/mp4\" \\\n  -F \"prompt=$(cat assets/example_t2v_prompt.json)\" \\\n  -F \"negative_prompt=$(cat assets/negative_prompt.json)\" \\\n  -F \"size=1280x720\" \\\n  -F \"num_frames=189\" \\\n  -F \"fps=24\" \\\n  -F \"num_inference_steps=35\" \\\n  -F \"guidance_scale=6.0\" \\\n  -F \"max_sequence_length=4096\" \\\n  -F \"flow_shift=10.0\" \\\n  -F \"seed=123\" \\\n  --output output.mp4\n```\n\nImage-to-video uses the same endpoint with an `input_reference` file part.\nAudio is generated jointly and muxed into the output MP4 (48 kHz stereo AAC) —\nthere is no separate audio endpoint.\n\n## Reasoner mode (multimodal understanding)\n\nThe same checkpoint serves a text/multimodal **Reasoner** for world\nunderstanding, future prediction, and action reasoning via the standard `vllm`\npackage (not vLLM-Omni):\n\n```bash\n# CUDA 13 drivers:\nuv pip install --torch-backend=cu130 \"vllm==0.21.0\" \\\n  \"vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3\" \\\n  openai\n# CUDA 12.8 drivers: use --torch-backend=cu128 \"vllm==0.19.1\"\n\nCUDA_VISIBLE_DEVICES=0 \\\nvllm serve nvidia/Cosmos3-Nano \\\n  --hf-overrides '{\"architectures\": [\"Cosmos3ReasonerForConditionalGeneration\"]}' \\\n  --tensor-parallel-size 1 \\\n  --mm-encoder-tp-mode data \\\n  --async-scheduling \\\n  --allowed-local-media-path / \\\n  --media-io-kwargs '{\"video\": {\"num_frames\": -1}}' \\\n  --port 8000\n```\n\nThen query the OpenAI-compatible `/v1/chat/completions` endpoint with text +\nimage/video content.\n\n## References\n\n- [Model card](https://huggingface.co/nvidia/Cosmos3-Nano)\n- [vLLM-Omni](https://github.com/vllm-project/vllm-omni)\n- [cosmos-framework](https://github.com/NVIDIA/cosmos-framework)\n- [Cosmos](https://github.com/nvidia/cosmos)\n",
  "source": {
    "source": "vllm-omni",
    "repo": "vllm-project/vllm-omni",
    "ref": "main",
    "sha": "cda4a1ef83b295ba332146a7709e6277725edc9f",
    "path": "recipes/models/nvidia/Cosmos3-Nano.yaml",
    "edit_url": "https://github.com/vllm-project/vllm-omni/edit/main/recipes/models/nvidia/Cosmos3-Nano.yaml",
    "overlays": []
  }
}