Tags: datajoint/datajoint-python
Tags
fix(#1442): scan_*_references reads raw JSON metadata, not decoded co… …dec output scan_hash_references and scan_schema_references called table.to_arrays(attr_name), which routes through decode_attribute and returns the codec's decoded payload (numpy.ndarray, NpyRef, ObjectRef, bytes, or local path str). None of those satisfy _extract_*_refs's `isinstance(value, dict) and "path" in value` check, so both helpers silently returned empty reference sets and gc.collect() would have classified live data as orphaned. Replace with table.proj(attr_name).cursor(as_dict=True). The cursor yields the raw JSON column value: a dict on PostgreSQL/JSONB or a JSON string on MySQL — both already handled by _extract_*_refs (gc.py:138 string branch, gc.py:145 dict branch). Backend-agnostic, custom-codec-safe, and turns scan into a metadata-only operation (no more downloading every external blob to discard the deserialized result). Also registers gc in _lazy_modules (src/datajoint/__init__.py) so dj.gc.scan(...) works as documented in the gc module docstring and in how-to/garbage-collection.md, matching the existing diagram entry pattern. Test scaffolding adds TestScanWithLiveData with three non-mocked e2e tests covering structurally distinct decoded-value types (<blob@> → ndarray, <npy@> → NpyRef, <object@> → ObjectRef). The same tests fail on the buggy version, proving the regression. The 26 existing mocked tests stay intact for orchestration coverage. GC remains non-transaction-safe (TOCTOU window between scan and delete); a two-phase quarantine→grace→purge API is the right remedy and will be tracked as a separate enhancement.
PreviousNext