fix(backtest): binomial significance gate on §9.6 verdict (rebased) #1051

Merged

bot-backend merged 1 commit from fix/backtest-sig-gate into main

2026-06-04 11:43:42 +00:00

bot-backend commented

2026-06-04 11:43:39 +00:00

Collaborator

Re-based replacement for #1050 (which was accidentally cut from a stale base missing #1048's variants → unmergeable). Now correctly on top of #1048 + #1049.

Adds a statistical-significance gate to the read-only §9.6 backtest verdict so a high OOS hit-rate on a thin window can't be over-claimed.

_binom_sf_ge(k,n,p=0.5) exact one-sided binomial (stdlib math.comb); _VERDICT_ALPHA=0.05.
verdict() + cross_source_verdict() now require scorable AND lag_stable AND hit_rate>=0.5+margin AND P(X>=hits|n_test,0.5) < α.
The Source-A Almon 6/10 (0.60) near-miss now correctly reads keep advisory — not significant (p=0.38). Ground-truth confirmed no live signal (full-sample R²≈0.003, wrong sign).
Verdict text states n_test + binomial p on pass and fail.

Tests: 99 passed (union of #1048 variant tests + binomial gate tests incl. the 6/10 regression guard). ruff clean. code-review APPROVE (binomial math executed/verified). Refs #978.

Re-based replacement for #1050 (which was accidentally cut from a stale base missing #1048's variants → unmergeable). Now correctly on top of #1048 + #1049. Adds a **statistical-significance gate** to the read-only §9.6 backtest verdict so a high OOS hit-rate on a thin window can't be over-claimed. - `_binom_sf_ge(k,n,p=0.5)` exact one-sided binomial (stdlib `math.comb`); `_VERDICT_ALPHA=0.05`. - `verdict()` + `cross_source_verdict()` now require scorable AND lag_stable AND `hit_rate>=0.5+margin` AND `P(X>=hits|n_test,0.5) < α`. - The Source-A Almon **6/10 (0.60)** near-miss now correctly reads *keep advisory — not significant (p=0.38)*. Ground-truth confirmed no live signal (full-sample R²≈0.003, wrong sign). - Verdict text states n_test + binomial p on pass and fail. Tests: 99 passed (union of #1048 variant tests + binomial gate tests incl. the 6/10 regression guard). ruff clean. code-review APPROVE (binomial math executed/verified). Refs #978.

bot-backend added 1 commit 2026-06-04 11:43:39 +00:00

fix(backtest): add binomial significance gate to §9.6 verdict

Deploy / changes (push) Successful in 5s

Details

Deploy / build-frontend (push) Has been skipped

Details

Deploy / build-backend (push) Successful in 28s

Details

Deploy / build-worker (push) Successful in 28s

Details

Deploy / deploy (push) Successful in 58s

Details

cc66f51863

The OOS verdict flagged a variant 'candidate to promote' on hit-rate >= 0.5+margin
+ lag_stable alone. On thin data this over-claims: Source A Almon-ADL scored 6/10
(0.60) lag-stable and was flagged as signal, but P(X>=6|10,0.5)~=0.377 -- a coin
flip. Live ground-truth confirmed no signal (full-sample R2~=0.003, wrong sign).

Add exact stdlib-only one-sided binomial _binom_sf_ge + _VERDICT_ALPHA=0.05 and
require P(X>=hits|n_test,0.5) < alpha in both verdict() and cross_source_verdict()
on top of the effect-size margin. hits recovered exactly as round(hit_rate*n_test)
(n_test==scored invariant; no evaluator shape change). Verdict text now states
n_test + the binomial p on pass and fail. Evaluator/estimator math and the
read-only SELECT discipline untouched. Refs #978.