feat(backtest): validate #978 Almon-ADL + #979 seasonal as OOS variants #1048

Merged

bot-backend merged 1 commit from feat/backtest-oos-almon-seasonal into main

2026-06-04 09:56:36 +00:00

bot-backend commented

2026-06-04 09:55:58 +00:00

Collaborator

What

Extends the read-only §9.6 rate-sensitivity OOS backtest harness with two opt-in candidate-method variants so #978/#979 wiring is evidence-based, not blind.

--almon (#978): new evaluate_oos_almon — Almon polynomial distributed-lag (regression.fit_almon_dl), fit on TRAIN only, point-in-time sum_j beta_j*drate[t-j] test prediction.
--deseasonalize (#979): train-only month-of-year factors (normalize.seasonal_factors) divided out before log_diff, then the existing best_lag evaluator.

Both pin the fit to _time_ordered_split(n_train) — no look-ahead leakage (adversarial unit tests assert the train fit is byte-identical when only the test window is corrupted). Default path (best_lag/raw) is byte-identical to before. Harness stays SELECT-only and is not in the prod image.

88 tests pass, ruff clean. Deep-code-review: APPROVE (leakage/point-in-time/backward-compat independently verified).

Prod OOS findings (directional hit-rate; coin-flip 0.50; bar 0.55 + lag-stable)

Variant	EKB-wide OOS	lag-stable	verdict
B raw best_lag (baseline)	0.148	no	no signal
B deseasonalized (#979)	0.148	no	no change
B Almon-ADL (#978)	0.407	yes	3x baseline, sub-bar
A raw best_lag	0.400	no	no signal
A deseasonalized (#979)	0.400	no	no change
A Almon-ADL (#978)	0.600	yes	SIGNAL > coin-flip+margin

#979 deseasonalize: validated NEGATIVE — no OOS lift anywhere. Keep advisory.
#978 Almon-ADL: validated POSITIVE — dominates best_lag on every tier, fixes lag-instability; on survivorship-free Source A clears the promotion bar (0.60). Source B (more data) 0.407 — better but sub-bar. Source A thin (~13mo) caveat applies.

Wiring of #978 into the §9.6 consumer is a separate follow-up decision (this PR only adds the validation instrument).

## What Extends the **read-only** §9.6 rate-sensitivity OOS backtest harness with two opt-in candidate-method variants so #978/#979 wiring is **evidence-based, not blind**. - `--almon` (#978): new `evaluate_oos_almon` — Almon polynomial distributed-lag (`regression.fit_almon_dl`), fit on TRAIN only, point-in-time `sum_j beta_j*drate[t-j]` test prediction. - `--deseasonalize` (#979): train-only month-of-year factors (`normalize.seasonal_factors`) divided out before `log_diff`, then the existing `best_lag` evaluator. Both pin the fit to `_time_ordered_split(n_train)` — **no look-ahead leakage** (adversarial unit tests assert the train fit is byte-identical when only the test window is corrupted). Default path (`best_lag`/raw) is byte-identical to before. Harness stays SELECT-only and is not in the prod image. **88 tests pass, ruff clean.** Deep-code-review: APPROVE (leakage/point-in-time/backward-compat independently verified). ## Prod OOS findings (directional hit-rate; coin-flip 0.50; bar 0.55 + lag-stable) | Variant | EKB-wide OOS | lag-stable | verdict | |---|---|---|---| | B raw best_lag (baseline) | 0.148 | no | no signal | | B deseasonalized (#979) | 0.148 | no | no change | | B Almon-ADL (#978) | 0.407 | **yes** | 3x baseline, sub-bar | | A raw best_lag | 0.400 | no | no signal | | A deseasonalized (#979) | 0.400 | no | no change | | A Almon-ADL (#978) | **0.600** | **yes** | **SIGNAL > coin-flip+margin** | - **#979 deseasonalize: validated NEGATIVE** — no OOS lift anywhere. Keep advisory. - **#978 Almon-ADL: validated POSITIVE** — dominates `best_lag` on every tier, fixes lag-instability; on survivorship-free Source A clears the promotion bar (0.60). Source B (more data) 0.407 — better but sub-bar. Source A thin (~13mo) caveat applies. Wiring of #978 into the §9.6 consumer is a separate follow-up decision (this PR only adds the validation instrument).

bot-backend added 1 commit 2026-06-04 09:55:59 +00:00

feat(backtest): validate #978 Almon-ADL + #979 seasonal as OOS variants

Deploy / changes (push) Successful in 6s

Details

Deploy / build-frontend (push) Has been skipped

Details

Deploy / build-backend (push) Successful in 32s

Details

Deploy / build-worker (push) Successful in 33s

Details

Deploy / deploy (push) Successful in 59s

Details

692f468010

Extend the read-only §9.6 rate-sensitivity OOS harness with two opt-in
candidate-method variants so any wiring decision is evidence-based:
- --almon: evaluate_oos_almon, Almon distributed-lag (regression.fit_almon_dl),
  fit on TRAIN only, point-in-time sum_j beta_j*drate[t-j] prediction.
- --deseasonalize: train-only month-of-year factors (normalize.seasonal_factors)
  divided out before log_diff, then the existing best_lag evaluator.
Both pin the fit to _time_ordered_split(n_train); no look-ahead leakage
(adversarial tests assert the train fit is byte-identical under test corruption).
Default path (best_lag/raw) is byte-identical to before. 88 tests pass, ruff clean.

Prod OOS findings (directional hit-rate, coin-flip 0.50, bar 0.55+lag-stable):
- #979 deseasonalize: neutral (B 0.148->0.148, A 0.40->0.40) -> keep advisory.
- #978 Almon-ADL: dominates best_lag (B 0.148->0.407 lag-stable; A 0.40->0.60,
  clears coin-flip+margin) -> candidate to promote from advisory.