Skip to content

[DQ Threshold W3.2] (follow-up) minRowsPerDimension exemption for small dimension groups #33241

Description

@TeddyCr

Deliberately deferred — ship ANY_DIMENSION first and find out whether anyone actually hits the flapping.

The concern

A dimension group with 3 rows and 1 violation is 33%, so ANY_DIMENSION + PERCENTAGE flaps on long-tail dimensions.

How it would work

  1. Add a minRowsPerDimension NUMBER parameter, default 0 (no exemption — nothing changes unless asked).
  2. In _process_dimension_rows (base_test_handler.py:412) each dimension row already carries total_count, so no extra query is needed.
  3. A group below the minimum is still evaluated, still displayed, still gets its own status and impact score — it is only excluded from the ANY_DIMENSION roll-up.
  4. The UI marks such groups so an exempted failing group is visible rather than silently ignored.

Scope of applicability — narrower than it first looks

  • Only meaningful when dimensionFailurePolicy = ANY_DIMENSION. Under OVERALL_ONLY nothing rolls up.
  • Only meaningful when thresholdUnit = PERCENTAGE. Under ABSOLUTE there is no small-sample distortion — 1 violation in a 3-row group is 1 violation, a legitimate comparison against "tolerate 5 rows".
  • It does not help with the Others bucket, whose total_count is large by construction. Tail dilution is a separate, unsolved limitation.

Triage before building: it is a third knob on an already-busy form, and ANY_DIMENSION is itself opt-in.

Depends on the dimensional roll-up sub-task.

Design spec: docs/plans/2026-09-11-dq-failure-thresholds.md. Parent epic: #16301.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

  • Status
    Release Backlog 🚧

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions