NUMERICAL AUDIT · 1 NOVEMBER 2026

confidenceinterval: three formula errors in the delta-method intervals for F1, precision and recall

The analytic confidence intervals in confidenceinterval use a wrong multinomial covariance, wrong macro precision/recall derivatives and a mistranscribed macro F1 variance; nominal 95% macro-precision intervals cover about 86%.

Xamit Kadirbekov, Daniyal Kadirbekov · AI-assisted investigation and writing

Checked 1 November 2026: PR #17 is open, with no review yet; not merged or released. Local regression checks and limitations documented.

Upstream report #17

Authors: Xamit Kadirbekov and Daniyal Kadirbekov (GERO). Published 1 November 2026.

Status: Checked 1 November 2026: PR #17 is open, with no review yet; not merged or released. Local regression checks and limitations documented.

The analytic confidence intervals in confidenceinterval use a wrong multinomial covariance, wrong macro precision/recall derivatives and a mistranscribed macro F1 variance; nominal 95% macro-precision intervals cover about 86%.

Published proposal

• jacobgil/confidenceinterval PR #17 — head commit 688d8bcb6a6f6b99b0ca6b607b519b65362ca7be, OPEN at submission on 1 November 2026.

What the functions do

confidenceinterval is a small Python package (146 GitHub stars, about 250 downloads a month when checked) that attaches confidence intervals to classifier metrics, including for diagnostic models. Precision of a class is the share of correct cases among those the model assigned to it; recall (sensitivity) is the share found among those that truly belong to it; F1 is their harmonic mean; "macro" averages over classes. The package computes analytic intervals with the delta method: Var ≈ gᵀ Σ g / n, where g is the gradient of the metric and Σ = diag(p) − p·pᵀ is the multinomial covariance of the confusion-matrix proportions. The package's own docstrings state this formula. We checked commit 5dd3e58121fb, identical to version 1.0.5 on PyPI.

The discrepancies

1. Multinomial covariance (takahashi_methods.py, lines 95, 278, 426): np.diag(p) - p * p.transpose(). For a 1-D array .transpose() does nothing, so p² is subtracted element-wise instead of the outer product. Affects binary F1 and macro precision and recall.
2. Macro precision/recall derivatives (lines 88 and 271): the diagonal derivatives are not divided by the column sum (precision) or row sum (recall). The code's own off-diagonal derivatives are divided correctly.
3. Macro F1 variance (lines 469–480): the formula of Takahashi et al. (2022, Applied Intelligence 52:4961) is transcribed with + instead of ×, with b_i + b_j instead of b_i·b_j, and with counts instead of proportions.

Check against the paper's Table 1 (n = 100, confusion matrix [[2,2,2],[5,70,2],[0,2,15]], rows = prediction):

On the paper's example the error happens to be small. Elsewhere it is not:

• Macro precision on [[30,5,5],[10,20,10],[2,8,10]]: library (0.502, 0.645), reference (0.477, 0.670). In a Monte Carlo of 400 replications the nominal 95% interval covered the truth about 86% of the time for macro precision and 88% for macro recall; the reference delta method covered 95% and 96%.
• Macro F1 on [[50,10],[20,20]]: (0.506, 0.834) against (0.574, 0.766).
• Over 1,600 random confusion matrices the macro F1 SE ranged from 0.47× to 4.2× the correct value, and some cases returned NaN. The error goes both ways: too wide on many tables, too narrow when one class is rare.
• The package README shows a macro F1 CI of (0.548, 1.113), whose upper end is above 1.

Reference and why this is not a convention

The reference treats each metric as a function of the table proportions, takes the gradient numerically in mpmath and uses Σ = diag(p) − ppᵀ; it does not call the library. It reproduces the paper's published SE of 0.0650 exactly. We are checking the method the library itself names, not a competing one. A no-op transpose on a 1-D array is a NumPy slip, not a formula choice, and the missing division contradicts the code's own off-diagonal derivatives. The paper's macro F1 formula, as extracted from PubMed Central, is garbled, so we re-derived it with the delta method and confirmed it against the paper's number.

Fix and tests

• np.outer(p, p) in the three covariance matrices; diagonal derivatives of macro precision/recall divided by the column/row sum; macro F1 variance re-transcribed on proportions.
• New tests/test_takahashi_variance.py compares every analytic SE with the numerical delta method and includes the paper's Table 1 example. Full suite: 8 failed / 27 passed before, 35 passed after.

Upstream status

As of 1 November 2026, PR #17 is open, with no review or comment. Nothing has been merged or released. The author's last commit is from May 2024, so a quick review is not expected.

Limits

The point estimates of the metrics are correct; only the analytic intervals are affected. Micro averaging, the proportion intervals delegated to statsmodels and the bootstrap intervals are not affected. We did not check how often the package is cited or used in published studies. Issue #2 asked for the paper's macro F1 variance; this PR fixes its transcription. The open PR #16 (DeLong in float32) belongs to another author and is unrelated. Novelty rests on a read of all 16 issues and PRs; forks and copies were not checked.

The paper example was re-computed in a separate session on 1 November before submission. No reproduction archive is attached to this page yet.

Attribution and license

AI tools (Claude) helped find the cases, build the reference calculations and draft the patches and tests; the examples were re-run independently before submission. Xamit Kadirbekov and Daniyal Kadirbekov are the report authors and reviewed the submitted diff; upstream software remains credited to its developers. Report: CC BY 4.0. Upstream patches retain their upstream licenses.