Graceful shutdown: flush foyer NVMe writer and drain in-flight requests #13

Problem

When the process receives SIGTERM (e.g. during a container restart or deployment), there is no coordinated shutdown sequence. Two things can go wrong:

1. foyer NVMe corruption. foyer's hybrid cache has a background writer that flushes entries from RAM to the NVMe tier asynchronously. If the process exits mid-flush, the NVMe cache directory can be left in an inconsistent state. On restart, foyer may fail to recover those entries or — worse — serve corrupted data.

2. Dropped in-flight requests. Active HTTP requests (especially long-running ones like cold queries against S3 at 25s p50) are aborted mid-stream, returning connection-reset errors to clients.

Proposal

Wire up tokio::signal to catch SIGTERM / SIGINT and run a shutdown sequence:

1. Stop accepting new connections — axum's Server::with_graceful_shutdown handles this.
2. Drain in-flight requests — wait for active handlers to complete, with a configurable timeout (e.g. FIRNFLOW_SHUTDOWN_TIMEOUT_SECS, default 30s). Requests that exceed the timeout are dropped.
3. Close the foyer HybridCache — call .close() which flushes the NVMe write buffer and syncs the directory. This must happen after request draining so no new entries are being written during the flush.

Considerations

• The 202-async endpoints (warmup, index build, compaction) spawn background tokio::tasks. On shutdown, these should be cancelled or allowed to complete within the drain timeout. Index builds can take 200+ seconds, so cancellation is more practical than waiting.
• A JoinSet or CancellationToken pattern for tracking background tasks would make this cleaner.
• Log a message at shutdown indicating how many requests were drained and whether the cache flush succeeded.

References

• axum graceful shutdown: axum::serve::Serve::with_graceful_shutdown
• foyer HybridCache::close(): flushes pending writes to NVMe
• Current main.rs: crates/firnflow-api/src/main.rs

Further considerations

• Tie shutdown semantics to the operation registry from Add operation status tracking for background API work #34 once it lands. Running operations should transition to a documented terminal state on shutdown — failed, abandoned, or cancelled — so callers polling an operation see something coherent rather than a dangling running record. Pick the wording deliberately and document it in the same change.
• In a future clustered deployment, a worker lease (also from Add operation status tracking for background API work #34) lets another replica pick up abandoned operations after lease expiry. Express that boundary now so the shutdown path doesn't hardcode "running → failed" and make later recovery awkward.