Skip to content

rag: teardown lock-ordering residuals (gap-collection D↔K, user-delete U↔T) need a serialization point #1389

Description

@OchnikBartek

What breaks

#1387 gives every teardown path one lock order - the COLLECTION_TEARDOWN
advisory lock before any knowledge_bases/organizations row lock - which
removes the claim vs purge ABBA. Two deadlock residuals survive it, both
needing a serialization point (an outbox, or a generation table) that this
lock-ordering pass cannot express. Both are narrow, and #1387 made both
strictly narrower than they were - but neither is closed.

1. Gap-collection D↔K (a purge residual, narrowed by #1387).
OrganizationService.purge takes teardown locks for a snapshot of the org's
collections read without the org row lock (organization.py:384), then the org
FOR UPDATE (:389), then the authoritative re-list and delete (:392-395,
docs D before the KB row K, as ON DELETE SET NULL + #1116 require). A
collection committed in the window between the snapshot and the org lock is in
the authoritative list but not in held, so purge deletes its D then K
under the org lock without ever holding its teardown lock.

Reachable only for a collection created in the microsecond snapshot→lock
window inside an org being deleted, then immediately deleted-again or
reclaimed - and self-resolving (Postgres aborts one side; the caller retries).
Before #1387 the teardown lock was taken late everywhere, so every
org-scoped collection had this exposure; now only gap collections do.

2. User-delete U↔T (pre-existing, untouched by #1387).
UserService.delete takes the subject's users row FOR UPDATE first
(user.py:397_lock_for_delete :402, for #1115/#1134), then
_purge_personal_collections takes each personal collection's teardown lock
(:552-558) - order U → T. A concurrent create of a personal KB owned by
that user takes the teardown lock in claim (collection_access.py:231) and
then, on INSERT, FOR KEY SHARE on the owner's users row - order T → U.
Sharing a collection name (#913) crosses them → 40P01. This inversion is in
the users dimension, which #1387 addressed only in the organizations one;
it predates #1364.

Where

  • backend/app/services/organization.py:382-414 - the snapshot, the org lock,
    the D → K authoritative delete, and the held-gated reserve.
  • backend/app/services/user.py:397,402,440 (U FOR UPDATE) and :552-560
    (_purge_personal_collections teardown locks).
  • backend/app/services/collection_access.py:231 - claim takes T before the
    create INSERT that FK-locks the users/organizations row.

How you'd know it's fixed

One serialization point so teardown and a concurrent write never take
{teardown, org/user/kb-row} in opposing orders, including for a collection
born mid-teardown and including the users dimension: a concurrent
create-and-teardown of a same-named collection never 40P01s and never adopts a
table the teardown then drops, and a user-delete racing a personal-KB create of
a same-named collection never 40P01s. The constraint that makes both
non-trivial is the one #1382 already weighed and declined as disproportionate:
purge must hold the org row lock across its deletes (SET NULL FK + CHECK,
#1115), and the write path holds the teardown lock across the org/user-FK
insert, so a lock reorder alone cannot close either - only a generation/outbox
that removes the cross-order lock pair.

Split from #1382, found reviewing #1378 (#1387's adversarial review). Cluster:
RAG teardown / "Ingestion and retrieval" in the issue map #168 (siblings #1355,
#1362, #1364, #1382, #1387).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingseverity:lowClean-up, dead code, minor duplication

Type

Projects

  • Status
    Done

Relationships

None yet

Development

No branches or pull requests

Issue actions