llm-codebox

A lightweight code-execution sandbox that exposes a FastAPI service backed by a Jupyter Kernel Gateway and Redis (Valkey). The current codebase is adapted from https://github.com/ChenShawn/MultiModal-Jupyter-Sandbox.

flowchart TB
    subgraph app["Application Layer"]
        main["main.py<br/>FastAPI App"]
        api["api.py<br/>API Endpoints"]
        config["config.py"]
        logging["logging_setup.py"]
    end
    
    subgraph jupyter["jupyter/ Package"]
        init["__init__.py<br/>Public API"]
        manager["manager.py<br/>KernelSessionManager"]
        sessions["sessions.py<br/>SessionRegistry"]
        channels["channels.py<br/>KernelChannels"]
        executor["executor.py<br/>JupyterExecutor"]
        gateway["gateway_http.py<br/>KernelGatewayClient"]
        persistence["state_persistence.py<br/>StatePersistence"]
        models["models.py<br/>ExecuteResult"]
        types["types.py<br/>Protocols & TypedDicts"]
        internal["internal_states.py<br/>Dump/Load Code"]
    end
    
    subgraph external["External Services"]
        redis[(Redis/Valkey)]
        kernel[Jupyter Gateway]
    end
    
    main --> api
    main --> config
    main --> logging
    api --> init
    init --> manager
    manager --> sessions
    manager --> channels
    manager --> executor
    sessions --> gateway
    sessions --> persistence
    persistence --> internal
    persistence --> executor
    executor --> channels
    executor --> models
    channels --> kernel
    gateway --> kernel
    persistence --> redis
    sessions --> types

• Jupyter Kernel Gateway: Executes Python code (with GPU support)
• Redis/Valkey: Stores evicted session state for restoration

Default setup uses the Jupyter Kernel Gateway. If you need scalable compute, deploy your own enterprise gateway and point KERNEL_GATEWAY_URL to it.

docker compose up --build

The API will be available at http://localhost:12345.

The jupyter container is configured to use all available NVIDIA GPUs. Prerequisites:

1. NVIDIA driver installed on host
2. Install nvidia-container-toolkit:

   sudo apt install nvidia-container-toolkit
   sudo systemctl restart docker

To use specific GPUs, edit docker-compose.yaml:

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: 2              # use 2 GPUs
          # or
          device_ids: ["0", "2"] # use specific GPUs
          capabilities: [gpu]

The Jupyter container comes with common data science packages pre-installed (numpy, pandas, matplotlib, scikit-learn, torch, etc.). See jupyter-gateway/requirements.txt for the full list.

Execute Python code with structured output (separates result, stdout, stderr, images).

Request body:

{
  "session_id": "optional-session-id",
  "code": "x = 42\nprint('hello')\nx",
  "timeout": 5.0
}

Response body:

{
  "output": {
    "result": "42",
    "stdout": "hello\n",
    "stderr": "",
    "images": ["iVBORw0KGgo..."],
    "debug_info": {"msg": ""}
  },
  "status": "success",
  "execution_time": 0.123
}

Execute Python code with combined output format.

Request body:

{
  "session_id": "optional-session-id",
  "code": "print('hello')",
  "timeout": 5.0
}

Response body:

{
  "output": {
    "stdout": "hello\n42\n",
    "stderr": "",
    "stdout_raw": "hello\n42\n",
    "stderr_raw": "",
    "images": ["data:image/png;base64,iVBORw0KGgo..."]
  },
  "status": "success",
  "execution_time": 0.123
}

Notes:

• session_id defaults to JupyterSandboxDefault if omitted.
• timeout defaults to 5.0 seconds.

Clear a kernel session and delete its marker from Redis.

Request body:

{
  "session_id": "optional-session-id"
}

Response body:

{
  "status": "success"
}

Health check endpoint.

Response body:

{
  "status": "ok"
}

List all active and persisted sessions.

Response body:

{
  "active_sessions": 2,
  "max_sessions": 16,
  "sessions": [
    {
      "session_id": "abc123",
      "kernel_id": "kernel-uuid-1",
      "status": "active",
      "idle_seconds": 45.2
    },
    {
      "session_id": "xyz789",
      "kernel_id": null,
      "status": "persisted",
      "idle_seconds": null
    }
  ]
}

Environment variables used by the FastAPI service:

• KERNEL_GATEWAY_URL (required): Base URL for the Jupyter Kernel Gateway.
• KERNEL_GATEWAY_TOKEN (optional): Token for the Kernel Gateway auth header.
• REDIS_URL (optional): Full Redis connection URL.
• REDIS_HOST (default: valkey)
• REDIS_PORT (default: 6379)
• REDIS_DB (default: 0)
• PORT (default: 12345)
• MAX_CONCURRENT_RUNS (default: 4): Maximum concurrent code executions.
• MAX_ACTIVE_SESSIONS (default: MAX_CONCURRENT_RUNS): Maximum active kernel sessions. When exceeded, LRU session is evicted to Redis.

The provided docker-compose.yaml sets defaults for local usage.

1. Ensure you have a running Jupyter Kernel Gateway and Redis-compatible server.
2. Set KERNEL_GATEWAY_URL and Redis settings.
3. Run:

uvicorn llm_codebox.main:app --host 0.0.0.0 --port 12345