NUMERICAL AUDIT · 1 NOVEMBER 2026

toad: PSI drops bins that empty out, and IV ignores the missing-value group

toad's PSI silently skips a bin that is empty in one sample, so a vanished 10% segment reads as stable, and its IV scores the NaN group of numeric features with substituted counts.

Xamit Kadirbekov, Daniyal Kadirbekov · AI-assisted investigation and writing

Checked 1 November 2026: PR #164 is open and awaiting review; not merged or released. Local regression checks and limitations documented.

Upstream report #164

Authors: Xamit Kadirbekov and Daniyal Kadirbekov (GERO). Published 1 November 2026.

Status: Checked 1 November 2026: PR #164 is open and awaiting review; not merged or released. Local regression checks and limitations documented.

toad's PSI silently skips a bin that is empty in one sample, so a vanished 10% segment reads as stable, and its IV scores the NaN group of numeric features with substituted counts.

Published proposal

• amphibian-dev/toad PR #164 — head commit 335b796b8130bed41bb4cb69bc498e5e68df2465, OPEN at submission on 1 November 2026.

What the functions do

toad is an open-source toolkit for building credit scorecards: weight-of-evidence binning, Information Value (IV), the Population Stability Index (PSI) and feature selection. We checked commit 4317ecef972d68b82db15a17ed0c031cf10fdcca, version 0.1.7; its metrics.py and stats.py are byte-identical to the PyPI wheel.

• PSI tells a bank whether the distribution of a feature or score has moved since the model was built: PSI = Σ (current share − base share) · ln(current share / base share), summed over bins. Common thresholds (Siddiqi, Credit Risk Scorecards): below 0.10 stable, 0.10–0.25 watch, above 0.25 significant shift.
• IV measures how differently "bad" and "good" customers are spread across a feature's groups; it is used to select features, and toad.selection.select(iv=...) filters on it.

The PSI discrepancy

_PSI (toad/metrics.py) subtracts two value_counts(normalize=True) Series with different indexes. A bin missing from one side becomes NaN, and the pandas sum silently skips it. The bin that disappeared completely, the strongest possible change, contributes nothing.

import pandas as pd, toad
base = pd.Series(list("A"*45 + "B"*45 + "C"*10))
test = pd.Series(list("A"*50 + "B"*50))      # category C (10% of base) disappeared
toad.metrics.PSI(test, base)                 # 0.010536  -> "stable"
# = 2*(0.5-0.45)*ln(0.5/0.45): the C term is simply missing

The second example uses the documented path toad.metrics.PSI(test, base, combiner=c). A new category that did not exist in the base is lost in the same way.

The IV discrepancy

_IV (toad/stats.py) selects each group with feature == v. For v = NaN this is always False, so the missing group is scored with the substituted default counts (one bad, one good) instead of the real customers. This affects numeric features with few distinct values (at most 10, the CONTINUOUS_NUM threshold). The same data passed as strings gives the correct value.

import numpy as np, pandas as pd, toad
x = np.array([0., 0., 1., 1., np.nan, np.nan, np.nan, np.nan]); y = np.array([0, 1, 0, 1, 1, 1, 1, 0])
toad.stats.IV(x, y)                            # 0.2043
toad.stats.IV(pd.Series(x).astype(object), y)  # 0.2930  (= hand calculation)

On a simulated feature with 2,000 customers, values 0–3 and 25% missing, where the missing group has a 30% default rate against 5% elsewhere, toad gives IV 0.334 against a reference of 0.814.

Reference and why this is not a convention

The PSI reference is the definition summed over the union of bins, computed with exact fractions; the IV reference is a hand calculation from the IV definition and toad's own result on the same data as strings. Packages legitimately differ in how they smooth empty bins (1e-4, 0.9 records, 1e-6). No convention drops the bin, and toad's documentation does not say it does. For IV, the same data giving different values depending on column dtype is an internal inconsistency.

Fix and tests

• _PSI aligns both proportion Series on the union of bins with fill_value=0; an empty bin now gives inf, as the definition requires. The PR offers to switch to a smoothed PSI if the maintainers prefer; the point is that the bin must not be dropped.
• _IV selects the missing group with pd.isnull.
• New tests test_PSI_bin_empty_in_test, test_PSI_bin_new_in_test, test_PSI_frame_keeps_empty_bin, test_iv_nan_group_uses_real_counts; all fail before and pass after.
• toad/metrics_test.py and toad/stats_test.py: 5 failed / 28 passed before, 1 failed / 32 passed after. The remaining failure is the existing test_KS, which compares floats exactly (0.055367756612569874 != 0.05536775661256989) and fails the same way without the change.

Upstream status

As of 1 November 2026, PR #164 is open with review required and no review yet. Nothing has been merged or released.

Limits

We did not measure how often a bin empties out in any real portfolio. Novelty rests on a GitHub search and a read of the titles of all 125 issues; _PSI has not changed since October 2019. Forks and non-GitHub forums were not searched. The return_frame=True table already shows NaN in the empty cell; the summary number is what omits it.

The PSI and IV examples were re-run in a separate session on 1 November before submission. No reproduction archive is attached to this page yet.

Attribution and license

AI tools (Claude) helped find the cases, build the reference calculations and draft the patches and tests; the examples were re-run independently before submission. Xamit Kadirbekov and Daniyal Kadirbekov are the report authors and reviewed the submitted diff; upstream software remains credited to its developers. Report: CC BY 4.0. Upstream patches retain their upstream licenses.