Declare the contract
Batch operations act on independent examples along the last feature axis. Check partitioning, permutation, duplication and rank changes. Reductions across the batch have a different contract.
f(concat(x, y)) ≈ concat(f(x), f(y))PRECISION IS A PROPERTY. TEST IT.
Confidence, down to
the last decimal.
An open workspace for the numerical behavior of ML systems.
Inspect the contract. Compare the outputs. Reproduce the result.
THE COMPARISON01 / ONNX
ONNX RuntimeReading environment…
ACTUAL|actual − reference| ≤ atol + rtol · |reference|ONNX ReferenceEvaluatorOfficial implementation
REFERENCEEvaluations—Loading real run data
Within contract—Agreement + invariant checks
Backend divergences—Candidates for investigation
Contract / execution issues—Baseline not assessed
EVIDENCE WORKSPACE
Find the signal.
Loading run…↓ Run report
| Operation / test case | Input | Optimization | Max abs. error | Result | Details |
|---|---|---|---|---|---|
| Loading measured results… | |||||
i
A difference is a starting point. A backend disagreement does not identify which implementation is wrong. A precision regression requires a compatible baseline; upstream bug status requires investigation.
Read the criteria ↗Precision across operations
Observed outcomes, grouped by operation. Every segment links to the underlying evaluations.
PassedDivergenceContract issueExecution / unsupported
Inference time, with context
Median of per-case medians, milliseconds. Includes Python call overhead; excludes session construction.
The reference evaluator is a correctness implementation, so these timings are not a production speed ranking.
| Operation | Measured pairs | Runtime median | Reference median |
|---|
Keep the mathematics honest
Probabilities must be finite and strictly within [0, 1]. Their sum uses a dtype-specific tolerance. A fully masked row returns exactly zero. LayerNorm variance includes epsilon.
Var(normalized) ≈ v / (v + ε)Compare independent paths
Run identical graphs in ONNX Runtime and official ReferenceEvaluator. Check algebraic graph variants and analytic NumPy float64 formulas. Agreement alone is insufficient; matching NaN outputs fail.
NaN == NaN is never a passing resultPreserve the evidence
Download the original graph, inputs and SHA-256 manifest. Observations retain backend versions, output hashes, exact tolerances and session settings. Repeated identical observations are deduplicated.
python -m gero_stability.cli replay ARTIFACT_DIRA bounded, reproducible scope.
This build measures CPU execution on seeded synthetic graphs using opset 19 / IR 10. QuantizeLinear and DynamicQuantizeLinear are excluded. DequantizeLinear remains covered. CUDA, model-level accuracy and global batch metrics are outside this run.
Float64 formulas are diagnostic checks, not arbitrary-precision proofs. A passing finite sample does not certify every possible input.
Default comparison tolerances
| Dtype | rtol | atol |
|---|---|---|
| float16 | 5e−3 | 5e−4 |
| float32 | 1e−5 | 1e−6 |
| float64 | 1e−12 | 1e−13 |
Each observation stores its actual tolerance, including CLI overrides.