Flexible KV cache reuse for knowledge-intensive LLM serving

Built on vLLM and LMCache. Designed for compute-network-aware knowledge injection across LLM systems.

Why CacheRoute? • Features • Architecture • Frontend URLs • Quick Start • API • Docs

CacheRoute is a lightweight LLM scheduling framework built on vLLM and LMCache to enable flexible KV cache reuse across LLM systems. It targets knowledge-intensive LLM services, such as browser AI and knowledge QA systems, where many requests repeatedly use the same external knowledge. Existing systems usually prepend long knowledge texts to the user question and send the whole prompt to the model for recomputation. Although this approach helps reduce model hallucination and improve answer quality, it introduces heavy prefill overhead and redundant computation when the same knowledge appears across many requests.

CacheRoute addresses this problem by using dedicated servers to store KVCache blocks for popular knowledge. For each request, CacheRoute dynamically chooses between text-based injection and KVCache-based injection according to task queues, compute load, and network load. CacheRoute therefore shifts knowledge-injection cost between compute and network resources, improving task latency and system throughput.

• 🚀 Less redundant prefill computation: reuse repeated knowledge through KV cache instead of recomputing long prompts.
• 🔁 Cross-system KV cache reuse: share reusable knowledge across LLM systems through KDN servers.
• 🌐 Compute-network coordination: dynamically choose between recomputation and KV cache injection based on real-time resource load.

CacheRoute reduces average TTFT, improves system throughput, and enables more effective KVCache reuse under knowledge-intensive workloads.

CacheRoute separates global routing, local injection decisions, and KV cache management into Scheduler, Proxy, Instance, and KDN Server components.

• Scheduler: performs global resource-pool selection and knowledge-oriented task routing.
• Proxy: manages local task queues, selects the knowledge-injection strategy, and exposes the main Proxy browser UI.
• Instance: connects CacheRoute to vLLM + LMCache and handles execution signaling.
• KDN Server: stores reusable knowledge and injects KVCache blocks when needed.
• Resource Agent/Dashboard: optionally observes local Instance resource snapshots for validation and future control-plane integration.

The URLs above assume a single-machine deployment with loopback addresses. Containers without host networking must publish the corresponding ports or use reachable host addresses.

1. The Client sends an OpenAI-compatible request to the Scheduler.
2. The Scheduler analyzes the knowledge requirement and selects a target resource pool.
3. The Proxy predicts the cost of text-based and KVCache-based injection.
4. The KDN Server injects reusable KVCache blocks when KVCache reuse is selected.
5. The Instance forwards the request to vLLM + LMCache and returns the response.
6. Instance resource snapshots can flow through the Proxy control plane. The Proxy aggregates a compact pool_resource snapshot and reports it to the Scheduler through registration and heartbeat payloads.
7. The optional Proxy UI and Instance Resource Dashboard visualize control-plane and resource state for debugging and validation.

CacheRoute has been tested with the following core environment:

Install CacheRoute's application dependencies with:

python3 -m pip install -r requirements.txt
python3 -m pip check

requirements.txt intentionally does not pin PyTorch, vLLM, or LMCache. These packages belong to the serving image and must remain compatible with its CUDA environment.

For the complete Docker, source-build, Rust, Tkinter, X11, Redis, and LMCache setup, use env/README.md as the deployment source of truth.

CacheRoute provides two ways to get started.

Set USE_MOCK = True in core/config.py, then start the demo components in separate terminals. The main demo entrypoints can be executed directly from test/; they add the repository root to Python's module search path automatically.

cd test

python3 demo_scheduler.py --cacheroute
python3 demo_kdn.py
python3 demo_proxy.py \
  --strategy round_robin \
  --injection-strategy iws \
  --ready-release-policy text_bypass
python3 demo_instance.py --port 9001 --host 127.0.0.1
python3 demo_client.py --with-ui

demo_proxy.py starts the Proxy browser UI at a URL similar to http://127.0.0.1:8202. demo_client.py --with-ui starts the Client UI at http://127.0.0.1:7071/ui/client.

The recommended host/container paths are:

Host repository:      /llm-stack/CacheRoute
Container repository: /workspace/llm-stack/CacheRoute

When a compatible complete image already exists, create the headless/browser-mode container with:

sudo docker run --gpus all -it \
  --name cacheroute-main \
  --network host \
  --ipc=host \
  --shm-size=64g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --memory=0 \
  --memory-swap=0 \
  -v /llm-stack:/workspace/llm-stack \
  -w /workspace/llm-stack \
  cacheroute:vllm0.13-lmcache3.11-pytorch2.9.1 \
  bash

Use env/README.md when building the image, installing PyTorch/vLLM/LMCache from source, enabling Tkinter/X11, repairing an older image, or recreating a deleted container.

DEFAULT_MODEL = "/workspace/llm-stack/models/LLM-Research/Meta-Llama-3-70B-Instruct"
DEFAULT_MODEL_SHORTNAME = "llama3-70b"
EMBEDDING_MODEL = "/workspace/llm-stack/CacheRoute/model/embedder/intfloat__multilingual-e5-large-instruct"
USE_MOCK = False

cd /workspace/llm-stack/CacheRoute
python3 -m pip install -r requirements.txt
python3 -m pip check

sudo docker run -d \
  --name lmcache-redis \
  --network host \
  redis:7 \
  redis-server \
    --bind 0.0.0.0 \
    --protected-mode no \
    --save "" \
    --appendonly no \
    --maxmemory 200gb \
    --maxmemory-policy allkeys-lru

sudo docker start lmcache-redis

sudo docker exec -it lmcache-redis redis-cli ping
# Expected: PONG

mkdir -p /workspace/llm-stack/config

cat > /workspace/llm-stack/config/lmcache_with_redis.yaml <<'EOF'
chunk_size: 256
pre_caching_hash_algorithm: "sha256_cbor"

local_cpu: true
max_local_cpu_size: 80.0

remote_url: "redis://127.0.0.1:6379"
remote_serde: "cachegen"

local_disk: null
max_local_disk_size: 0

save_decode_cache: false
cache_policy: "LRU"
numa_mode: null
EOF

sudo docker exec -it cacheroute-main bash

export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export MODEL_DIR=/workspace/llm-stack/models/LLM-Research/Meta-Llama-3-70B-Instruct
export LMCACHE_CONFIG_FILE=/workspace/llm-stack/config/lmcache_with_redis.yaml
export PYTHONHASHSEED=0
export OMP_NUM_THREADS=8

python3 -m vllm.entrypoints.openai.api_server \
  --model "$MODEL_DIR" \
  --served-model-name llama3-70b \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.75 \
  --dtype auto \
  --max-model-len 4096 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 16384 \
  --kv-offloading-backend lmcache \
  --kv-offloading-size 64 \
  --disable-hybrid-kv-cache-manager \
  --kv-cache-metrics

curl -sS http://127.0.0.1:8000/v1/models | python3 -m json.tool

sudo docker exec -it cacheroute-main bash
cd /workspace/llm-stack/CacheRoute/test

python3 demo_scheduler.py \
  --cacheroute \
  --kdn-pending-overload-th 8 \
  --kdn-active-overload-th 4 \
  --kdn-queue-ms-overload-th 30 \
  --cacheroute-log-decision 1

sudo docker exec -it cacheroute-main bash
cd /workspace/llm-stack/CacheRoute/test
python3 demo_kdn.py

sudo docker exec -it cacheroute-main bash
cd /workspace/llm-stack/CacheRoute
mkdir -p data/quickstart

cat > data/quickstart/cacheroute.txt <<'EOF'
CacheRoute is a knowledge-oriented LLM scheduling framework. It reduces repeated prefill computation by storing and reusing KVCache blocks for frequently requested external knowledge. The Scheduler selects a KDN server and a Proxy, while the Proxy chooses between text recomputation and KVCache reuse.
EOF

python3 kdn_server/kdn_register_cli.py

:buildkv_file /workspace/llm-stack/CacheRoute/data/quickstart/cacheroute.txt --api-url http://127.0.0.1:8000/v1/chat/completions --model llama3-70b
:pool

curl -sS http://127.0.0.1:7001/debug/status | python3 -m json.tool

sudo docker exec -it cacheroute-main bash
cd /workspace/llm-stack/CacheRoute/test

python3 demo_proxy.py \
  --strategy round_robin \
  --injection-strategy iws \
  --ready-release-policy text_bypass

http://127.0.0.1:8202

sudo docker exec -it cacheroute-main bash
cd /workspace/llm-stack/CacheRoute/test

python3 demo_instance.py \
  --host 127.0.0.1 \
  --port 9001 \
  --proxy-cp-url http://127.0.0.1:8002

curl -sS http://127.0.0.1:7001/debug/status | python3 -m json.tool
curl -sS http://127.0.0.1:8002/v1/instance/list?include_dead=true | python3 -m json.tool

sudo docker exec -it cacheroute-main bash
cd /workspace/llm-stack/CacheRoute/test
python3 demo_client.py --with-ui

http://127.0.0.1:7071/ui/client

python3 demo_client.py

curl http://127.0.0.1:7001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3-70b",
    "messages": [
      {
        "role": "user",
        "content": "What does CacheRoute reuse to reduce repeated prefill computation?"
      }
    ],
    "temperature": 0,
    "max_tokens": 64,
    "stream": false,
    "RAG": true
  }'

curl -sS http://127.0.0.1:7001/debug/strategy | python3 -m json.tool

Client/curl
  -> Scheduler :7001
  -> selected Proxy :8001
  -> Instance :9001
  -> vLLM + LMCache :8000
  -> response returned through the same chain

See kdn_server/README.md for detailed knowledge registration and KVCache injection commands, and core/README.md for multi-machine configuration.

Build-check the Rust resource agent:

cargo check --manifest-path instance/resource_agent/Cargo.toml

Start the Tkinter desktop dashboard only when the image contains python3.12-tk and the container was created with X11 forwarding as documented in env/README.md:

python3 instance/resource_dashboard/dashboard_app.py \
  --agent-listen 127.0.0.1:9201 \
  --sample-interval-ms 1000 \
  --instance-id hp_127.0.0.1:9001

Use the browser fallback in headless environments:

python3 instance/resource_dashboard/dashboard_server.py \
  --dashboard-listen 0.0.0.0:9202 \
  --agent-listen 127.0.0.1:9201

Open:

http://127.0.0.1:9202

The resource dashboard is a validation helper and does not change Scheduler, Proxy, Instance, or KDN behavior.

CacheRoute exposes OpenAI-compatible API endpoints through the Scheduler.

curl http://127.0.0.1:7001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3-70b",
    "messages": [{"role": "user", "content": "What is DeepSeek"}],
    "max_tokens": 1,
    "stream": false,
    "RAG": true
  }'

curl http://127.0.0.1:7001/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3-70b",
    "prompt": "What is DeepSeek",
    "max_tokens": 1,
    "RAG": true
  }'

CacheRoute is under active development. The current release supports:

• Scheduler-side knowledge-oriented routing.
• KDN selection based on knowledge coverage and overload filtering.
• Proxy selection based on topology, load safety window, and knowledge history.
• Proxy-side dynamic injection strategy selection.
• KDN-based text registration and KVCache registration.
• Proxy browser UI for control-plane, topology, Instance liveness, and resource-snapshot observability.
• Optional Instance resource snapshots through a Rust agent and dashboard.
• Debugging APIs such as /debug/status and /debug/strategy.

Suggested minimum validation commands:

cd test
python3 demo_scheduler.py --cacheroute
curl -s http://127.0.0.1:7001/debug/status
curl -s http://127.0.0.1:7001/debug/strategy

• Scheduler-side knowledge-oriented routing
• Proxy-side dynamic injection strategy selection
• KDN-based text and KVCache registration
• OpenAI-compatible request forwarding
• Proxy browser observability UI
• Optional Instance resource dashboard
• Scheduler browser UI
• KDN Server browser UI
• More deployment examples
• Benchmark scripts and reproducible evaluation
• More KV cache placement policies
• Paper and citation release