Light1YT
1f490b7915
Add scripts/backtest_rate_sensitivity.py — STRICTLY READ-ONLY out-of-sample validation gate for the §9.6 engine (#1009). Monthly sold-units from Source B (objective_lots) EKB-wide + per class, aligned to monthly key_rate, time-ordered holdout (fit oldest 70%, test newest), point-in-time OOS directional hit-rate + in-sample-vs-OOS honesty block + per-tier lift + coin-flip+lag-stability verdict. Deterministic, no LLM, no DB writes. CLI --since/--holdout-frac/--classes/--json. 36 unit tests on synthetic. VERDICT (run vs prod 2026-06-03): EKB-wide OOS hit-rate 0.148 (< 0.5 coin-flip), lag unstable → §9.6 engine NOT OOS-validated → stays ADVISORY. Likely confounded by Source B survivorship bias (recent-month inflation → spurious trend); clean re-validation needs Source A depth (13mo now, deepens weekly) or survivorship correction. The harness re-validates when data improves. |
||
|---|---|---|
| .. | ||
| api | ||
| integration | ||
| scrapers | ||
| scripts | ||
| services | ||
| smoke | ||
| sql | ||
| workers | ||
| __init__.py | ||
| test_admin_weight_profiles.py | ||
| test_concepts_stub.py | ||
| test_gate_verdict.py | ||
| test_health.py | ||
| test_layout_signature.py | ||
| test_layout_tz_pdf.py | ||
| test_nspd_client.py | ||
| test_nspd_sync.py | ||
| test_ping.py | ||
| test_poi_score.py | ||
| test_quarter_dump_lookup.py | ||
| test_rbac.py | ||
| test_sentry_init.py | ||
| test_velocity.py | ||
| test_weight_profiles.py | ||