NUMERICAL AUDIT · 1 NOVEMBER 2026

popmon: the strongest KS drift becomes NaN and is shown green

A missing branch in popmon's port of ROOT's Kolmogorov probability turns very large KS scores into NaN, which the default traffic light reports as green; PSI and JSD are also computed on raw counts.

Xamit Kadirbekov, Daniyal Kadirbekov · AI-assisted investigation and writing

Checked 1 November 2026: PR #299 is open, with no review yet; not merged or released. Local regression checks and limitations documented.

Upstream report #299

Authors: Xamit Kadirbekov and Daniyal Kadirbekov (GERO). Published 1 November 2026.

Status: Checked 1 November 2026: PR #299 is open, with no review yet; not merged or released. Local regression checks and limitations documented.

A missing branch in popmon's port of ROOT's Kolmogorov probability turns very large KS scores into NaN, which the default traffic light reports as green; PSI and JSD are also computed on raw counts.

Published proposal

ing-bank/popmon PR #299 — head commit 89a612ab7e33af31f09302a976c360d8d9457d40, OPEN at submission on 1 November 2026.

What the function does

popmon, maintained under ING's GitHub organisation, builds data-stability reports over time. For every feature and every period it compares the histogram with a reference and colours the result with a traffic light. Monitoring model inputs and scores, for example in credit-risk models, is the kind of use it is built for.

For the Kolmogorov–Smirnov (KS) comparison, popmon takes the largest gap D between two cumulative distributions, converts it to z = D·√(N·M/(N+M)), turns z into a p-value with ks_prob, and reports a z-score −Φ⁻¹(p). The default rule "*_zscore": [7, 4, -4, -7] marks a z-score of 7 or more red and 4 or more yellow; everything else is green.

The discrepancy

ks_prob (popmon/analysis/comparison/comparisons.py) says in its docstring that it translates ROOT's TMath::KolmogorovProb without modifying the formulas. ROOT ends with a branch that returns 0 for u ≥ 6.8116, where the probability is below 10⁻⁴⁰. popmon initialises the p-value to NaN and has no such branch, so for large scores it returns NaN. The z-score becomes NaN, and traffic_light compares NaN with every bound, gets False each time and returns green. The existing test pins ks_prob(10.0) == nan.

from popmon.analysis.comparison.comparisons import ks_prob
from scipy.special import kolmogorov
for z in [6.8, 6.9, 10.0, 48.0]:
    print(z, ks_prob(z), kolmogorov(z))
# 6.8   1.37e-40   1.37e-40
# 6.9   nan        8.86e-42
# 10.0  nan        2.77e-87
# 48.0  nan        0.0

The threshold is easy to reach on large samples: with N = M = 100,000, z = 6.81 corresponds to a gap D of only 0.03.

End-to-end run through pm_stability_report (popmon 1.4.12, commit af654436f9c4195cc105735169d5e0595fe5b16e). Reference: four weeks of 20,000 scores each from N(600, 50).

Current week	KS z	popmon p-value	z-score	popmon traffic light	Reference p
no drift	1.34	0.056	1.59	green	0.056
no drift, 5,000 rows	0.52	0.95	−1.63	green	0.95
shift +0.1σ (5 points)	5.67	2.3·10⁻²⁸	10.98	red	2.3·10⁻²⁸
shift +1σ (50 points)	48.3	NaN	NaN	green	0

The small shift is flagged red; the shift ten times larger is shown green. The comparison with the previous period (prev1_ks_zscore) behaves the same way.

Second issue: PSI and JSD on counts

population_stability_index, kullback_leibler_divergence and jensen_shannon_divergence are defined on proportions (their own unit tests pass proportions), but the histogram comparer passes them raw bin counts. The value then depends on sample sizes. In the same run, two weeks without drift gave a "PSI" of 83,670 and 209,296, and the +1σ week gave 135,531; computed on the proportions of the same bins the values are 0.005, 0.015 and 0.988. JSD: 9,659 / 19,968 / 13,301 against 0.0006 / 0.0018 / 0.110.

The default PSI traffic light is built from deviations of the metric across periods, not from the raw value, and in our run all PSI/JSD lights were green. We do not claim this second issue changes a colour. It changes the reported number and any comparison of it with the usual 0.1/0.25 thresholds.

Reference and why this is not a convention

The reference is ROOT's documented formula (p = 0 for u ≥ 6.8116) and scipy.special.kolmogorov. Below the threshold popmon agrees with SciPy (z = 1: 0.26999967167737987 against 0.26999967167735456; z = 3, 6 and 6.8 identical), so the translation is correct except for the missing branch. The code itself cites ROOT and promises unmodified formulas. A p-value for such a gap is defined and is practically zero; NaN is not a meaningful alternative. PSI and JSD are defined on proportions.

Fix and tests

ks_prob returns 0 for u ≥ 6.8116, as ROOT does; the z-score is then +∞ and the light is red.
PSI, KL and JSD normalise their inputs to proportions before applying epsilon; for inputs that are already proportions nothing changes.
test_ks_prob now expects 0 at z = 10. New: test_ks_prob_large_score_is_zero_not_nan, test_ks_zscore_large_drift_is_red, test_psi_jsd_on_counts_equal_psi_jsd_on_proportions.
tests/popmon/analysis/comparison/test_comparisons.py: 4 failed / 5 passed before the fix, 9 passed after; 25/25 in the affected test directories in a local run. A wider run of tests/popmon/analysis and alerting had 15 environment-related failures (histogrammar / pandas 3) that were identical before and after.

Upstream status

As of 1 November 2026, PR #299 is open, with no review or comment. Nothing has been merged or released.

Limits

We do not know how many deployments rely on the KS traffic light with default settings, and no production monitoring system was tested. The examples are synthetic. Novelty rests on a bounded GitHub search of issues and PRs (ks, kolmogorov, nan, p-value, zscore, psi, jsd, normalisation) and the file history: ks_prob has not changed since the first commit of 27 April 2020. Forks and discussions outside GitHub were not checked. This is not an exhaustive priority claim.

The ks_prob comparison with SciPy was re-run in a separate session on 1 November before submission. No reproduction archive is attached to this page yet; the example above is self-contained.

Attribution and license

AI tools (Claude) helped find the cases, build the reference calculations and draft the patches and tests; the examples were re-run independently before submission. Xamit Kadirbekov and Daniyal Kadirbekov are the report authors and reviewed the submitted diff; upstream software remains credited to its developers. Report: CC BY 4.0. Upstream patches retain their upstream licenses.