D3: under sampling the threshold is evaluated on the sample, and that is surfaced loudly — no extrapolation, no blocking.
Provenance — reuse the existing mechanism, do not invent one
An earlier draft proposed an EvaluationScope value object. Not needed: RuntimeParameterSetter + get_runtime_parameters() (base_test_handler.py:708) already injects a typed model into a validator, and tableDiff already uses it to reach the table's profiler config (table_diff_params_setter.py:139 → models.py:49).
Add a setter carrying the resolved sample config + partition details and render it into the message.
Messages must carry the full picture
Today's "Found nullCount=3. It should be 0" does not say what it was measured against.
Found 120 null rows out of 9,981 evaluated (1.20%). Threshold is 1.00%, so this test failed.
Evaluated on a 10% sample of the table, partitioned on event_date >= 2026-09-10.
Mean of `amount` is 87.4. Expected between 90 and 110, widened by a 5% tolerance to 85.5-115.5,
so this test passed. Evaluated on the full table.
Always state the threshold actually applied — an older ingestion agent against a newer server ignores the parameter, and the message is the only way that becomes visible rather than inferred.
Notes
PERCENTAGE is sample-stable; ABSOLUTE is not (100 nulls in a 10% sample implies ~1,000).
- Statistical tests: location statistics (mean, median) survive sampling;
MIN/MAX are biased inward and SUM scales with the sample — a deviation threshold on columnValuesSumToBeBetween under sampling is close to meaningless.
- Rule-library SQL bypasses the sampler; say so.
- These strings are English-only free text in
TestCaseResult.result. The UI must re-derive its sentence from the parameters, never parse this string.
Acceptance
Depends on the row-tolerance and deviation sub-tasks.
Design spec: docs/plans/2026-09-11-dq-failure-thresholds.md. Parent epic: #16301.
D3: under sampling the threshold is evaluated on the sample, and that is surfaced loudly — no extrapolation, no blocking.
Provenance — reuse the existing mechanism, do not invent one
An earlier draft proposed an
EvaluationScopevalue object. Not needed:RuntimeParameterSetter+get_runtime_parameters()(base_test_handler.py:708) already injects a typed model into a validator, andtableDiffalready uses it to reach the table's profiler config (table_diff_params_setter.py:139→models.py:49).Add a setter carrying the resolved sample config + partition details and render it into the message.
Messages must carry the full picture
Today's
"Found nullCount=3. It should be 0"does not say what it was measured against.Always state the threshold actually applied — an older ingestion agent against a newer server ignores the parameter, and the message is the only way that becomes visible rather than inferred.
Notes
PERCENTAGEis sample-stable;ABSOLUTEis not (100 nulls in a 10% sample implies ~1,000).MIN/MAXare biased inward andSUMscales with the sample — a deviation threshold oncolumnValuesSumToBeBetweenunder sampling is close to meaningless.TestCaseResult.result. The UI must re-derive its sentence from the parameters, never parse this string.Acceptance
PERCENTAGEagrees within sampling error,ABSOLUTEdiffers, both messages state the populationDepends on the row-tolerance and deviation sub-tasks.
Design spec:
docs/plans/2026-09-11-dq-failure-thresholds.md. Parent epic: #16301.