Light1YT 1f490b7915 feat(forecasting): read-only OOS backtest of §9.6 rate-sensitivity (#978) ...

Add scripts/backtest_rate_sensitivity.py — STRICTLY READ-ONLY out-of-sample
validation gate for the §9.6 engine (#1009). Monthly sold-units from Source B
(objective_lots) EKB-wide + per class, aligned to monthly key_rate, time-ordered
holdout (fit oldest 70%, test newest), point-in-time OOS directional hit-rate +
in-sample-vs-OOS honesty block + per-tier lift + coin-flip+lag-stability verdict.
Deterministic, no LLM, no DB writes. CLI --since/--holdout-frac/--classes/--json.
36 unit tests on synthetic.

VERDICT (run vs prod 2026-06-03): EKB-wide OOS hit-rate 0.148 (< 0.5 coin-flip),
lag unstable → §9.6 engine NOT OOS-validated → stays ADVISORY. Likely confounded
by Source B survivorship bias (recent-month inflation → spurious trend); clean
re-validation needs Source A depth (13mo now, deepens weekly) or survivorship
correction. The harness re-validates when data improves.

2026-06-03 14:32:18 +05:00

0 lines

Python

Raw Blame History