Overview

Relevant source files

• CONTRIBUTING.md
• Cargo.lock
• Cargo.toml
• README.md
• README.zh-CN.md
• components/src/dynamo/frontend/frontend_args.py
• components/src/dynamo/frontend/main.py
• docs/backends/sglang/sglang-examples.md
• docs/components/kvbm/README.md
• docs/components/kvbm/kvbm-guide.md
• docs/contribution-guide.md
• docs/design-docs/architecture.md
• docs/design-docs/disagg-serving.md
• docs/development/backend-guide.md
• docs/fault-tolerance/request-migration.md
• docs/getting-started/building-from-source.md
• docs/getting-started/introduction.md
• docs/getting-started/local-installation.md
• docs/getting-started/quickstart.mdx
• docs/index.yml
• docs/reference/release-artifacts.md
• fern/components/CustomFooter.tsx
• fern/docs.yml
• fern/fern.config.json
• fern/main.css
• fern/translations/zh-CN/pages-dev/contribution-guide.md
• lib/bindings/c/src/lib.rs
• lib/bindings/kvbm/Cargo.lock
• lib/bindings/python/Cargo.lock
• lib/bindings/python/Cargo.toml
• lib/bindings/python/pyproject.toml
• lib/bindings/python/rust/lib.rs
• lib/bindings/python/rust/llm.rs
• lib/bindings/python/rust/llm/entrypoint.rs
• lib/bindings/python/rust/llm/kv.rs
• lib/bindings/python/src/dynamo/_core.pyi
• lib/bindings/python/src/dynamo/llm/__init__.py
• lib/bindings/python/tests/test_kv_bindings.py
• lib/kv-router/Cargo.toml
• lib/llm/Cargo.toml
• lib/llm/src/discovery/model.rs
• lib/llm/src/discovery/model_manager.rs
• lib/llm/src/discovery/watcher.rs
• lib/llm/src/discovery/worker_set.rs
• lib/llm/src/endpoint_type.rs
• lib/llm/src/entrypoint.rs
• lib/llm/src/entrypoint/input/common.rs
• lib/llm/src/entrypoint/input/grpc.rs
• lib/llm/src/entrypoint/input/http.rs
• lib/llm/src/http/service.rs
• lib/llm/src/http/service/health.rs
• lib/llm/src/http/service/service_v2.rs
• lib/llm/src/kv_router.rs
• lib/llm/src/kv_router/scheduler.rs
• lib/llm/src/kv_router/sequence.rs
• lib/llm/src/lib.rs
• lib/llm/src/local_model.rs
• lib/llm/src/model_card.rs
• lib/llm/src/model_type.rs
• lib/mocker/Cargo.toml
• lib/mocker/src/lib.rs
• lib/runtime/Cargo.toml
• lib/runtime/examples/Cargo.lock
• lib/runtime/examples/Cargo.toml
• lib/runtime/src/config.rs
• lib/runtime/src/instances.rs
• lib/runtime/src/logging.rs

NVIDIA Dynamo is a high-throughput, low-latency distributed inference framework for serving generative AI and reasoning models. It provides a unified orchestration layer that turns a cluster of GPUs into a coordinated inference system by supporting multiple backends (SGLang, TensorRT-LLM, vLLM, and TokenSpeed) with advanced routing, multi-tier KV caching, and disaggregated serving README.md34-38

Dynamo is built in Rust for performance and Python for extensibility, providing the orchestration layer above individual inference engines to maximize throughput and minimize latency README.md38 lib/bindings/python/pyproject.toml34-42

---

System Architecture

Dynamo implements a three-tier architecture with distinct layers for request handling, routing, and backend execution. It bridges the gap between high-level Python inference engines and high-performance Rust networking logic.

Three-Tier Architecture and Code Mapping

Sources: README.md34-38 Cargo.toml4-32 lib/llm/Cargo.toml61-68 lib/bindings/python/Cargo.toml50-55

---

Package Structure

Dynamo is distributed as two primary Python packages backed by a shared Rust core:

Package	Language	Purpose	Compiled Module
ai-dynamo-runtime	Rust (PyO3)	Distributed runtime primitives, networking, discovery	dynamo._core lib/bindings/python/Cargo.toml19
ai-dynamo	Python + Rust	LLM serving infrastructure, backend integrations	Includes ai-dynamo-runtime lib/bindings/python/pyproject.toml16

The project is organized into a Rust workspace and a Python component library Cargo.toml4-31 lib/bindings/python/Cargo.toml4-6

Source Tree Organization

graph TD
    subgraph Dynamo
        id1[dynamo/]
        id1 --> lib[lib/]
        id1 --> components[components/src/dynamo/]
        id1 --> deploy[deploy/]
    end

    subgraph lib
        lib --> runtime[runtime/]
        lib --> llm[llm/]
        lib --> kv_router[kv-router/]
        lib --> mocker[mocker/]
        lib --> parsers[parsers/]
        lib --> tokens[tokens/]
        lib --> memory[memory/]
        lib --> bindings_python[bindings/python/]
        lib --> bindings_kvbm[bindings/kvbm/]
    end

    subgraph components
        components --> frontend[frontend/]
        components --> vllm[vllm/]
        components --> sglang[sglang/]
        components --> trtllm[trtllm/]
        components --> planner[planner/]
    end

    subgraph deploy
        deploy --> helm[helm/]
        deploy --> operator[operator/]
    end

    style lib fill:#f9f,stroke:#333,stroke-width:2px
    style runtime fill:#ccf,stroke:#333,stroke-width:2px
    style llm fill:#ccf,stroke:#333,stroke-width:2px
    style kv_router fill:#ccf,stroke:#333,stroke-width:2px
    style mocker fill:#ccf,stroke:#333,stroke-width:2px
    style parsers fill:#ccf,stroke:#333,stroke-width:2px
    style tokens fill:#ccf,stroke:#333,stroke-width:2px
    style memory fill:#ccf,stroke:#333,stroke-width:2px
    style bindings_python fill:#ccf,stroke:#333,stroke-width:2px
    style bindings_kvbm fill:#ccf,stroke:#333,stroke-width:2px

    style components fill:#f9f,stroke:#333,stroke-width:2px
    style frontend fill:#ccf,stroke:#333,stroke-width:2px
    style vllm fill:#ccf,stroke:#333,stroke-width:2px
    style sglang fill:#ccf,stroke:#333,stroke-width:2px
    style trtllm fill:#ccf,stroke:#333,stroke-width:2px
    style planner fill:#ccf,stroke:#333,stroke-width:2px

    style deploy fill:#f9f,stroke:#333,stroke-width:2px
    style helm fill:#ccf,stroke:#333,stroke-width:2px
    style operator fill:#ccf,stroke:#333,stroke-width:2px

Sources: Cargo.toml4-32 lib/bindings/python/Cargo.toml50-55 lib/llm/Cargo.toml5-12 lib/bindings/python/pyproject.toml124-127

---

Key Features

Dynamo provides advanced capabilities for datacenter-scale inference README.md51-60:

Feature	Implementation	Description
Disaggregated Serving	PrefillRouter	Separates prefill and decode into independently scalable GPU pools README.md55
KV-Aware Routing	KvRouter	Routes requests based on worker load and KV cache overlap to eliminate redundant prefill README.md54
KV Block Manager (KVBM)	lib/kvbm-*	Offloads KV cache across GPU → CPU → SSD → remote storage Cargo.toml69-75
ModelExpress	modelexpress	Streams model weights GPU-to-GPU via NIXL/NVLink for faster cold-starts README.md27 lib/llm/Cargo.toml111-112
SLA-Based Planner	dynamo.planner	Autoscaler that profiles workloads and right-sizes pools to meet latency targets components/src/dynamo/planner/main.py
Flash Indexer	lib/kv-router	High-performance inter-node KV cache indexing for global routing lib/bindings/python/Cargo.lock27
Fault Tolerance	migration	Canary health checks and in-flight request migration README.md72
TokenSpeed Support	dynamo.tokenspeed	Day-0 support for LightSeek's TokenSpeed inference engine for agentic workloads docs/index.yml109-111

For details, see Key Concepts.

---

Backend Support Matrix

Dynamo turns standalone inference engines into a coordinated system README.md36-38 Support is extended across multiple high-performance backends.

Feature	SGLang	TensorRT-LLM	vLLM	TokenSpeed
Disaggregated Serving	✅	✅	✅	✅
KV-Aware Routing	✅	✅	✅	✅
SLA-Based Planner	✅	✅	✅	✅
KVBM	🚧	✅	✅	-
Multimodal	✅	✅	✅	-
Tool Calling	✅	✅	✅	✅

Sources: README.md61-72 Cargo.toml44-65 lib/llm/Cargo.toml61-68 docs/index.yml109-111

---

Core Components

The framework is built from several interconnected Rust and Python components. This section maps high-level concepts to their implementation.

Runtime Hierarchy and Messaging

Sources: lib/bindings/python/src/dynamo/_core.pyi54-67 lib/bindings/python/rust/lib.rs6-7 lib/runtime/Cargo.toml5-12 lib/llm/src/kv_router.rs249-250

Key Code Entities

Entity	Location	Role
DistributedRuntime	lib/bindings/python/src/dynamo/_core.pyi	The core runtime managing lifecycle, discovery, and communication lib/bindings/python/src/dynamo/_core.pyi54-67
KvRouter	lib/llm/src/kv_router.rs	Decides which worker should be used based on KV overlap lib/llm/src/kv_router.rs200-202
KvBM	lib/kvbm-engine	The block manager for multi-tier KV cache offloading Cargo.toml72
OpenAIPreprocessor	lib/llm/src/preprocessor.rs	Handles chat template application and tokenization lib/llm/Cargo.toml171-180
Flash Indexer	lib/llm/src/kv_router/indexer	Provides fast, global KV cache lookup across the cluster lib/llm/src/kv_router.rs60-61
ModelManager	lib/llm/src/discovery/model_manager.rs	Manages model loading, weight streaming, and local registration lib/llm/src/discovery/model_manager.rs

Sources: Cargo.toml44-75 lib/llm/Cargo.toml5-12 lib/runtime/Cargo.toml5-12 lib/bindings/python/Cargo.toml19-30 lib/bindings/python/src/dynamo/_core.pyi54-67 lib/llm/src/kv_router.rs200-202 lib/llm/src/kv_router.rs60-61

---

Getting Started

To begin using Dynamo, we recommend starting with our container-based deployments or setting up a local development environment.

• Quick Start: Run your first model using pre-built NGC containers README.md27
• Local Setup: Install the Rust toolchain and Python dependencies using maturin lib/bindings/python/Cargo.lock164-165
• Kubernetes: Deploy Dynamo using Helm charts and the Dynamo Operator docs/index.yml123-134

For details, see Getting Started.