fix(backtest): binomial significance gate on §9.6 verdict (anti false-promote) #1050

Closed

bot-backend wants to merge 1 commit from fix/backtest-verdict-significance-gate into main

bot-backend commented

2026-06-04 11:24:35 +00:00

Collaborator

What

Adds a statistical-significance gate to the read-only §9.6 rate-sensitivity OOS backtest verdict.

Why (a real near-miss): the verdict flagged a variant "candidate to promote" on oos_hit_rate >= 0.5+margin AND lag_stable alone. On thin data this over-claims — Source A Almon-ADL scored 6/10 (0.60), lag-stable, and was flagged as SIGNAL. But P(X≥6 | n=10, p=0.5) ≈ 0.377 — indistinguishable from a coin flip. Live ground-truth on the production module confirmed there is no signal (full-sample R²≈0.003, positive/wrong sign, estimates swing −9.8…+0.06 across slices). The over-claim nearly drove a bad wiring decision.

How

_binom_sf_ge(k, n, p=0.5) — exact one-sided binomial P(X≥k), stdlib-only (math.comb, no scipy).
_VERDICT_ALPHA = 0.05.
Both verdict() and cross_source_verdict() now require: scorable AND lag_stable AND hit_rate ≥ 0.5+margin AND _binom_sf_ge(hits, n_test) < α. Significance is a HARD gate on top of the effect-size margin (a thin-but-perfect or large-but-thin window can no longer sneak through).
hits = round(oos_hit_rate · n_test) (exact — n_test == scored invariant; no change to the deep-reviewed evaluator return shape).
Verdict text now states n_test + the binomial p on BOTH pass and fail.

Effect

The 6/10 Source A Almon case now correctly reads "keep advisory — not significant (p=0.38 ≥ 0.05)". The estimator/SELECT-only/variant-plumbing are untouched.

Tests

_binom_sf_ge known-value checks (0.377, 0.0107, 0.03125, 0.0625, guards).
Regression guard test_does_not_promote_six_of_ten_not_significant (encodes the near-miss).
Stricter-not-looser: old 0.9@n=3 and 0.75@n=12 promote-cases now assert NON-promote; promote test uses 0.71@n_test=35 (p≈0.008).
76 passed, ruff clean. code-review: APPROVE (binomial math executed + verified; gate confirmed to flip 6/10 → no signal).

Refs #978.

## What Adds a **statistical-significance gate** to the read-only §9.6 rate-sensitivity OOS backtest verdict. **Why (a real near-miss):** the verdict flagged a variant "candidate to promote" on `oos_hit_rate >= 0.5+margin AND lag_stable` alone. On thin data this over-claims — Source A Almon-ADL scored **6/10 (0.60)**, lag-stable, and was flagged as SIGNAL. But P(X≥6 | n=10, p=0.5) ≈ **0.377** — indistinguishable from a coin flip. Live ground-truth on the production module confirmed there is **no signal** (full-sample R²≈0.003, positive/wrong sign, estimates swing −9.8…+0.06 across slices). The over-claim nearly drove a bad wiring decision. ## How - `_binom_sf_ge(k, n, p=0.5)` — exact one-sided binomial P(X≥k), **stdlib-only** (`math.comb`, no scipy). - `_VERDICT_ALPHA = 0.05`. - Both `verdict()` and `cross_source_verdict()` now require: scorable **AND** lag_stable **AND** `hit_rate ≥ 0.5+margin` **AND** `_binom_sf_ge(hits, n_test) < α`. Significance is a HARD gate on top of the effect-size margin (a thin-but-perfect or large-but-thin window can no longer sneak through). - `hits = round(oos_hit_rate · n_test)` (exact — `n_test == scored` invariant; no change to the deep-reviewed evaluator return shape). - Verdict text now states n_test + the binomial p on BOTH pass and fail. ## Effect The 6/10 Source A Almon case now correctly reads **"keep advisory — not significant (p=0.38 ≥ 0.05)"**. The estimator/SELECT-only/variant-plumbing are untouched. ## Tests - `_binom_sf_ge` known-value checks (0.377, 0.0107, 0.03125, 0.0625, guards). - Regression guard `test_does_not_promote_six_of_ten_not_significant` (encodes the near-miss). - Stricter-not-looser: old 0.9@n=3 and 0.75@n=12 promote-cases now assert NON-promote; promote test uses 0.71@n_test=35 (p≈0.008). - 76 passed, ruff clean. code-review: APPROVE (binomial math executed + verified; gate confirmed to flip 6/10 → no signal). Refs #978.

bot-backend added 1 commit 2026-06-04 11:24:36 +00:00

fix(backtest): add binomial significance gate to §9.6 verdict 5530f472b3

The OOS verdict flagged a variant 'candidate to promote' on hit-rate >= 0.5+margin
+ lag_stable alone. On thin data this over-claims: Source A Almon-ADL scored 6/10
(0.60) lag-stable and was flagged as signal, but P(X>=6|10,0.5)~=0.377 -- a coin
flip. Live ground-truth confirmed no signal (full-sample R2~=0.003, wrong sign).

Add exact stdlib-only one-sided binomial _binom_sf_ge + _VERDICT_ALPHA=0.05 and
require P(X>=hits|n_test,0.5) < alpha in both verdict() and cross_source_verdict()
on top of the effect-size margin. hits recovered exactly as round(hit_rate*n_test)
(n_test==scored invariant; no evaluator shape change). Verdict text now states
n_test + the binomial p on pass and fail. Evaluator/estimator math and the
read-only SELECT discipline untouched. Refs #978.