What breaks
#1387 gives every teardown path one lock order - the COLLECTION_TEARDOWN
advisory lock before any knowledge_bases/organizations row lock - which
removes the claim vs purge ABBA. Two deadlock residuals survive it, both
needing a serialization point (an outbox, or a generation table) that this
lock-ordering pass cannot express. Both are narrow, and #1387 made both
strictly narrower than they were - but neither is closed.
1. Gap-collection D↔K (a purge residual, narrowed by #1387).
OrganizationService.purge takes teardown locks for a snapshot of the org's
collections read without the org row lock (organization.py:384), then the org
FOR UPDATE (:389), then the authoritative re-list and delete (:392-395,
docs D before the KB row K, as ON DELETE SET NULL + #1116 require). A
collection committed in the window between the snapshot and the org lock is in
the authoritative list but not in held, so purge deletes its D then K
under the org lock without ever holding its teardown lock.
Reachable only for a collection created in the microsecond snapshot→lock
window inside an org being deleted, then immediately deleted-again or
reclaimed - and self-resolving (Postgres aborts one side; the caller retries).
Before #1387 the teardown lock was taken late everywhere, so every
org-scoped collection had this exposure; now only gap collections do.
2. User-delete U↔T (pre-existing, untouched by #1387).
UserService.delete takes the subject's users row FOR UPDATE first
(user.py:397 → _lock_for_delete :402, for #1115/#1134), then
_purge_personal_collections takes each personal collection's teardown lock
(:552-558) - order U → T. A concurrent create of a personal KB owned by
that user takes the teardown lock in claim (collection_access.py:231) and
then, on INSERT, FOR KEY SHARE on the owner's users row - order T → U.
Sharing a collection name (#913) crosses them → 40P01. This inversion is in
the users dimension, which #1387 addressed only in the organizations one;
it predates #1364.
Where
backend/app/services/organization.py:382-414 - the snapshot, the org lock,
the D → K authoritative delete, and the held-gated reserve.
backend/app/services/user.py:397,402,440 (U FOR UPDATE) and :552-560
(_purge_personal_collections teardown locks).
backend/app/services/collection_access.py:231 - claim takes T before the
create INSERT that FK-locks the users/organizations row.
How you'd know it's fixed
One serialization point so teardown and a concurrent write never take
{teardown, org/user/kb-row} in opposing orders, including for a collection
born mid-teardown and including the users dimension: a concurrent
create-and-teardown of a same-named collection never 40P01s and never adopts a
table the teardown then drops, and a user-delete racing a personal-KB create of
a same-named collection never 40P01s. The constraint that makes both
non-trivial is the one #1382 already weighed and declined as disproportionate:
purge must hold the org row lock across its deletes (SET NULL FK + CHECK,
#1115), and the write path holds the teardown lock across the org/user-FK
insert, so a lock reorder alone cannot close either - only a generation/outbox
that removes the cross-order lock pair.
Split from #1382, found reviewing #1378 (#1387's adversarial review). Cluster:
RAG teardown / "Ingestion and retrieval" in the issue map #168 (siblings #1355,
#1362, #1364, #1382, #1387).
What breaks
#1387 gives every teardown path one lock order - the
COLLECTION_TEARDOWNadvisory lock before any
knowledge_bases/organizationsrow lock - whichremoves the
claimvspurgeABBA. Two deadlock residuals survive it, bothneeding a serialization point (an outbox, or a generation table) that this
lock-ordering pass cannot express. Both are narrow, and #1387 made both
strictly narrower than they were - but neither is closed.
1. Gap-collection D↔K (a purge residual, narrowed by #1387).
OrganizationService.purgetakes teardown locks for a snapshot of the org'scollections read without the org row lock (
organization.py:384), then the orgFOR UPDATE(:389), then the authoritative re-list and delete (:392-395,docs
Dbefore the KB rowK, asON DELETE SET NULL+ #1116 require). Acollection committed in the window between the snapshot and the org lock is in
the authoritative list but not in
held, so purge deletes itsDthenKunder the org lock without ever holding its teardown lock.
KnowledgeBaseService.delete/DELETE /rag/collections/{name}of that same collection takes
T → K → D; purge takesD → Kon the samerows. Both hold one of the collection's row locks and wait on the other →
40P01.to_dropbut not tombstone-reserved(
organization.py:409-414) - taking its teardown lock after the org lockwould reinvert rag: claim and org purge take the teardown and organizations locks in opposite order (ABBA deadlock) #1387 against
claim'sT → O. So the durable cleanup'sreference re-check is the only guard, and a
claimof the name in thecommit-to-drop window is not refused (
is_reservedis false), the rag: deferred vector-table drop leaks across tenants via name reuse (and a default-KB clear regression) #1362window reopened for this subset.
Reachable only for a collection created in the microsecond snapshot→lock
window inside an org being deleted, then immediately deleted-again or
reclaimed - and self-resolving (Postgres aborts one side; the caller retries).
Before #1387 the teardown lock was taken late everywhere, so every
org-scoped collection had this exposure; now only gap collections do.
2. User-delete U↔T (pre-existing, untouched by #1387).
UserService.deletetakes the subject'susersrowFOR UPDATEfirst(
user.py:397→_lock_for_delete:402, for #1115/#1134), then_purge_personal_collectionstakes each personal collection's teardown lock(
:552-558) - orderU → T. A concurrent create of a personal KB owned bythat user takes the teardown lock in
claim(collection_access.py:231) andthen, on INSERT,
FOR KEY SHAREon the owner'susersrow - orderT → U.Sharing a collection name (#913) crosses them →
40P01. This inversion is inthe
usersdimension, which #1387 addressed only in theorganizationsone;it predates #1364.
Where
backend/app/services/organization.py:382-414- the snapshot, the org lock,the
D → Kauthoritative delete, and theheld-gated reserve.backend/app/services/user.py:397,402,440(UFOR UPDATE) and:552-560(
_purge_personal_collectionsteardown locks).backend/app/services/collection_access.py:231-claimtakesTbefore thecreate INSERT that FK-locks the
users/organizationsrow.How you'd know it's fixed
One serialization point so teardown and a concurrent write never take
{teardown, org/user/kb-row}in opposing orders, including for a collectionborn mid-teardown and including the
usersdimension: a concurrentcreate-and-teardown of a same-named collection never
40P01s and never adopts atable the teardown then drops, and a user-delete racing a personal-KB create of
a same-named collection never
40P01s. The constraint that makes bothnon-trivial is the one #1382 already weighed and declined as disproportionate:
purgemust hold the org row lock across its deletes (SET NULL FK + CHECK,#1115), and the write path holds the teardown lock across the org/user-FK
insert, so a lock reorder alone cannot close either - only a generation/outbox
that removes the cross-order lock pair.
Split from #1382, found reviewing #1378 (#1387's adversarial review). Cluster:
RAG teardown / "Ingestion and retrieval" in the issue map #168 (siblings #1355,
#1362, #1364, #1382, #1387).