NUMERICAL AUDIT · 1 NOVEMBER 2026

NannyML: one value below the reference minimum inflates the estimated Wasserstein distance

In NannyML's estimated Wasserstein path an extra 1/n shifts the chunk CDF above 1, so a single low record adds a spurious distance of about two natural spreads on a long-tailed feature.

Xamit Kadirbekov, Daniyal Kadirbekov · AI-assisted investigation and writing

Checked 1 November 2026: PR #457 is open, with no review yet; not merged or released. Local regression checks and limitations documented.

Upstream report #457

Authors: Xamit Kadirbekov and Daniyal Kadirbekov (GERO). Published 1 November 2026.

Status: Checked 1 November 2026: PR #457 is open, with no review yet; not merged or released. Local regression checks and limitations documented.

In NannyML's estimated Wasserstein path an extra 1/n shifts the chunk CDF above 1, so a single low record adds a spurious distance of about two natural spreads on a long-tailed feature.

Published proposal

• NannyML/nannyml PR #457 — head commit 3e36e9a8ea75caa5b045a241341a7d5234f5c90f, OPEN at submission on 1 November 2026.

What the function does

NannyML monitors deployed models for data drift. For continuous features one of its methods is the Wasserstein distance W = ∫|F_reference(x) − F_chunk(x)| dx — roughly, how much work it takes to move one distribution onto the other. Data are monitored in chunks (a week, 5,000 rows); by default an alert fires when W leaves the mean ± 3 standard deviations of the reference chunks. When the reference set has at least 10,000 rows, the default auto mode does not compute W exactly but estimates it from a 10,000-bin histogram.

The discrepancy

In WassersteinDistance._calculate (nannyml/drift/univariate/methods.py) the estimated path shifts the binned chunk CDF by

data_cdf = data_cdf + amount_smaller   # amount_smaller = (n_smaller + 1) / len(data)

The + 1 accounts for a helper point (self._ref_min) appended only to compute the left-tail term. Inside the reference range the shift should be the share of chunk values left of that range, n_smaller / len(data). With the extra 1/n the chunk CDF exceeds 1, and the within-range term gains about (ref_max − ref_min)/n. It triggers as soon as one chunk value lies below the reference minimum. Before commit f362f5c (31 May 2024, part of PR #393) the shift was computed correctly.

# copy of the estimated path vs scipy.stats.wasserstein_distance
ref = rng.lognormal(10, 1, 50_000); d = rng.lognormal(10, 1, 500); d[0] = ref.min() * 0.99
# scipy exact: 3239.15
# current:     3846.64   (max chunk CDF 1.002)
# with fix:    3238.19
# same chunk without the low value: scipy 2906.77, current 2906.47 (matches)

Mean error of the estimate with one value just below the reference minimum and no drift (reference of 50,000 values):

On long-tailed features (income, loan amount, turnover) one record below the historical minimum moves W by 1.8–2.2 natural spreads with no real drift. Reference chunks are subsets of the reference and contain no value below its minimum, so the threshold does not absorb the error.

Reference and why this is not a convention

The reference is scipy.stats.wasserstein_distance, the exact W, on the same data. This is not a choice of approximation: without a low value the estimate matches the exact distance (2906.47 against 2906.77), and the shifted quantity is a CDF that exceeds 1. The same quantity was computed correctly before f362f5c.

Fix and tests

• Shift data_cdf by n_smaller / len(data); the left- and right-tail terms are unchanged.
• New test test_wasserstein_estimated_matches_exact_with_value_below_reference_min compares the estimate with scipy.stats.wasserstein_distance (rel 1e-3); it fails before and passes after.
• tests/drift/test_univariate_drift_methods.py: 22 passed and 1 failed before, 23 passed after, including the existing Wasserstein test.

Upstream status

As of 1 November 2026, PR #457 is open, with no review or comment. Nothing has been merged or released. The repository's last push before our check was in July 2025.

Limits

NannyML requires Python < 3.13 and heavy dependencies, so it was not installed in full. The real methods.py (commit 4c86ee350dc541cb244f965cf555b4b2a004ae99) was executed in a minimal harness of the package's own modules, with only the plotting figure class stubbed, together with the package's test file. The error table comes from a script copy of the formula. We do not claim that a default alert fires in any particular deployment; the error is shown relative to the natural spread. Novelty rests on a GitHub search for "wasserstein" (issue #165 and PRs #142, #148, #182, #393, none about this).

The comparison with SciPy was re-run in a separate session on 1 November before submission. No reproduction archive is attached to this page yet.

Attribution and license

AI tools (Claude) helped find the cases, build the reference calculations and draft the patches and tests; the examples were re-run independently before submission. Xamit Kadirbekov and Daniyal Kadirbekov are the report authors and reviewed the submitted diff; upstream software remains credited to its developers. Report: CC BY 4.0. Upstream patches retain their upstream licenses.