NUMERICAL AUDIT · 1 NOVEMBER 2026

optbinning: scorecard p-values with class_weight understate standard errors about twofold

optbinning's new Wald p-values leave class_weight out of the Hessian, so with a balanced 5% default rate a pure-noise variable looks significant in 34–44% of samples instead of 5%.

Xamit Kadirbekov, Daniyal Kadirbekov · AI-assisted investigation and writing

Checked 1 November 2026: PR #431 is open, with no review yet; not merged or released. Monte Carlo reference and limitations documented.

Upstream report #431

Authors: Xamit Kadirbekov and Daniyal Kadirbekov (GERO). Published 1 November 2026.

Status: Checked 1 November 2026: PR #431 is open, with no review yet; not merged or released. Monte Carlo reference and limitations documented.

optbinning's new Wald p-values leave class_weight out of the Hessian, so with a balanced 5% default rate a pure-noise variable looks significant in 34–44% of samples instead of 5%.

Published proposal

guillermo-navas-palencia/optbinning PR #431 — head commit b60d6cdc69f9b4848cde463268a1ddd76221526c, OPEN at submission on 1 November 2026.

What the function does

A credit scorecard is a logistic regression on binned features. Each coefficient has a standard error (SE); the Wald test turns coefficient/SE into a p-value, and a validator uses it to decide whether a variable stays in the model. optbinning added these p-values in PR #408 (closing issue #224), merged on 15 October 2026. We checked commit aed0ccc51f5a18708464c024e1b48b43a55837b0, version 1.0.0.

With a rare default (2–5%), users often fit LogisticRegression(class_weight="balanced"), which up-weights every default so that the two classes weigh the same. This re-weights the likelihood; it does not add observations.

The discrepancy

_wald_pvalues (optbinning/scorecard/scorecard.py) uses cov = inv(X' W X) with W = p(1−p), plus the penalty term. scikit-learn multiplies class_weight into the sample weights during fitting, but class_weight does not enter this Hessian. Even if it did, as a frequency weight the answer would still be wrong: for a re-weighted estimator the covariance is the sandwich H⁻¹·M·H⁻¹ (Huber 1967; White 1982).

Monte Carlo check: 5% default rate, two N(0,1) variables, the second pure noise, LogisticRegression(class_weight="balanced").

	Runs	True SD of the coefficient	Reported SE	Ratio	Noise "significant" at 5%
Run 1	400	0.0745	0.0387	0.52	34%
Run 2 (separate session)	200	0.0870	0.0394	0.45	43.5%
With the patch	—	—	0.0770	—	6.2%

Run 1 used n = 4,000 and logit = −3 + 0.8·x1; the SE column refers to x1, the significance column to the noise variable x2. Without class_weight the same function is calibrated (ratio 0.98; 7.0% ± 1.3% rejections), so the issue is specific to the weighting. Through the public API (Scorecard(...).table(style="detailed")) one demo coefficient barely moves (−0.988 unweighted, −0.996 balanced), while its "Std. Error" drops from 0.067 to 0.036: the model did not become more precise; the formula changed.

Reference and why this is not a convention

The primary reference is the Monte Carlo spread of the coefficient across independent samples from a known model; it does not depend on any covariance formula. The second is the sandwich formula, which matches it (ratio 1.03). PR #408 states that it inverts the Hessian of the function scikit-learn minimises and that, with an L2 penalty, p-values are approximate and biased towards non-significance. With class_weight the bias goes the other way, towards false significance, by about a factor of two, and is not documented.

Fix and tests

If the estimator has class_weight, per-sample class weights are computed with sklearn.utils.class_weight.compute_sample_weight, enter the Hessian, and the sandwich covariance is returned. sample_weight keeps its current handling.
New test test_pvalues_class_weight compares SEs from Scorecard.table with an independent sandwich calculation. tests/test_scorecard.py: 22 passed and 1 failed before, 23 passed after.

Upstream status

As of 1 November 2026, PR #431 is open, with no review or comment. Nothing has been merged or released.

Limits

The size of the effect is a Monte Carlo estimate (about ±1.1 percentage points on the rejection rate in run 1). The same balancing weights passed through sample_weight also gave a too-small SE in our run (0.0371, 33.8%), because the existing code treats sample weights as frequency weights; the PR leaves that interpretation unchanged and we make no claim about which interpretation a given user intends. The L2-penalty approximation acknowledged in #408 is a separate matter. We found no discussion of class_weight in #408, #224 or #413; #385 concerns class_weight in binning, a different code path. This is a bounded GitHub search, not an exhaustive novelty claim.

The Monte Carlo check was re-run in a separate session on 1 November (run 2 above). No reproduction archive is attached to this page yet.

Attribution and license

AI tools (Claude) helped find the cases, build the reference calculations and draft the patches and tests; the examples were re-run independently before submission. Xamit Kadirbekov and Daniyal Kadirbekov are the report authors and reviewed the submitted diff; upstream software remains credited to its developers. Report: CC BY 4.0. Upstream patches retain their upstream licenses.