3) #1997

Merged

lekss361 merged 3 commits from feat/1966-backtest-fixture-gate into main

2026-06-27 12:55:36 +00:00

lekss361 commented

2026-06-27 12:50:57 +00:00

Owner

Summary

PR 3/3 of #1966 (accuracy-loop → CI regression gate). Ships the hermetic, DB-free regression gate that measures the full estimator spine against frozen ground-truth.

Harness (scripts/backtest_estimator.py): --dump-fixture (capture) + replay_fixture (hermetic, ZERO DB) + --from-fixture/--update-baseline CLI (gzip-transparent). Two prod-parity fixes in _predict_full_spine: anchor pre-fetch now gated exactly like estimate_quality (estimate_same_building_anchor_enabled + area/address); the n_analogs < MIN_CANDIDATES skip dropped (prod surfaces any positive estimate at low confidence).
Frozen artifacts (tests/fixtures/): backtest_full_fixture.json.gz (277 prod deals; frozen _price_from_inputs inputs incl. the 3 DB callables' recorded outputs) + backtest_baseline.json (the diff-visible metrics).
Gate (tests/test_backtest_regression_gate.py): replays the fixture through the live spine, recomputes metrics, asserts == baseline within float tolerance. Auto-collected by the existing tradein pytest run.

What it measures (the audit hypothesis, now quantified)

overall MAPE 18.63% / bias −1.2% (centred), but per-segment:

segment	MAPE	bias	n
эконом	18.0%	+17.2%	120
комфорт	16.7%	−8.1%	104
бизнес	22.1%	−21.5%	46
элит	38.6%	−38.6%	6

range-coverage 55.2% → this is a relative regression gate, not the absolute ≥80% SLA (that stays deferred per the #1966 re-scope: ground-truth is data-blocked until ~Q4'26).

Design notes

Replay is order-based (FIFO) for the 3 frozen DB callables, not arg-keyed → platform-stable (verified Linux ≡ Windows bit-exact) and surfaces estimator drift as a clean metric diff, not a KeyError.
Gate semantics = golden-file. To change estimator behaviour, regenerate the baseline hermetically (--from-fixture <gz> --update-baseline, NO DB) and justify the per-segment deltas — the accuracy delta shows up right in the PR diff.
Fixture (3 MB gz) re-extracts only on ground-truth refresh; the per-change artifact is the 3 KB baseline. .pre-commit-config.yaml large-file hook narrowly excludes the fixture (not a CI/merge-gate rule).
Known limitation: logic inside the 3 frozen callables (asking→sold ratio / quarter-index queries) is baked into the recorded returns → not caught as drift; refreshed by re-extraction.

Test plan

pytest tests/test_backtest_{estimator,fixture_roundtrip,regression_gate}.py → 63 passed
Gate green on true baseline; tamper (бизнес MAPE 22.1→12.0) → fails with an actionable "regenerate baseline" message
Cross-platform: Linux-generated baseline reproduced bit-exact on Windows
Settings parity: 45/45 captured settings == config defaults (fixture self-consistent)
code-reviewer ✅ on the harness record/replay fidelity

Follow-up (separate PR — pipeline rule → human review)

The tradein pytest gate currently runs post-merge (deploy-tradein.yml, deploy-gating). A true pre-merge hard gate needs a new pull_request job in ci.yml (extend the changes filter + a backend-tests-tradein job) + a Required check in branch protection. That edits pipeline rules → opened separately for human review.

Refs #1966

## Summary PR 3/3 of #1966 (accuracy-loop → CI regression gate). Ships the hermetic, DB-free regression gate that measures the **full** estimator spine against frozen ground-truth. - **Harness** (`scripts/backtest_estimator.py`): `--dump-fixture` (capture) + `replay_fixture` (hermetic, ZERO DB) + `--from-fixture`/`--update-baseline` CLI (gzip-transparent). Two prod-parity fixes in `_predict_full_spine`: anchor pre-fetch now gated exactly like `estimate_quality` (`estimate_same_building_anchor_enabled` + area/address); the `n_analogs < MIN_CANDIDATES` skip dropped (prod surfaces any positive estimate at low confidence). - **Frozen artifacts** (`tests/fixtures/`): `backtest_full_fixture.json.gz` (277 prod deals; frozen `_price_from_inputs` inputs incl. the 3 DB callables' recorded outputs) + `backtest_baseline.json` (the diff-visible metrics). - **Gate** (`tests/test_backtest_regression_gate.py`): replays the fixture through the live spine, recomputes metrics, asserts `== baseline` within float tolerance. Auto-collected by the existing tradein pytest run. ## What it measures (the audit hypothesis, now quantified) overall MAPE **18.63%** / bias **−1.2%** (centred), but per-segment: | segment | MAPE | bias | n | |---|---|---|---| | эконом | 18.0% | **+17.2%** | 120 | | комфорт | 16.7% | −8.1% | 104 | | бизнес | 22.1% | **−21.5%** | 46 | | элит | 38.6% | **−38.6%** | 6 | range-coverage **55.2%** → this is a **relative** regression gate, not the absolute ≥80% SLA (that stays deferred per the #1966 re-scope: ground-truth is data-blocked until ~Q4'26). ## Design notes - Replay is **order-based (FIFO)** for the 3 frozen DB callables, not arg-keyed → platform-stable (verified Linux ≡ Windows **bit-exact**) and surfaces estimator drift as a clean metric diff, not a `KeyError`. - **Gate semantics = golden-file.** To change estimator behaviour, regenerate the baseline hermetically (`--from-fixture <gz> --update-baseline`, **NO DB**) and justify the per-segment deltas — the accuracy delta shows up right in the PR diff. - Fixture (3 MB gz) re-extracts only on ground-truth refresh; the per-change artifact is the 3 KB baseline. `.pre-commit-config.yaml` large-file hook **narrowly** excludes the fixture (not a CI/merge-gate rule). - **Known limitation:** logic *inside* the 3 frozen callables (asking→sold ratio / quarter-index queries) is baked into the recorded returns → not caught as drift; refreshed by re-extraction. ## Test plan - [x] `pytest tests/test_backtest_{estimator,fixture_roundtrip,regression_gate}.py` → **63 passed** - [x] Gate green on true baseline; tamper (бизнес MAPE 22.1→12.0) → fails with an actionable "regenerate baseline" message - [x] Cross-platform: Linux-generated baseline reproduced **bit-exact** on Windows - [x] Settings parity: 45/45 captured settings == config defaults (fixture self-consistent) - [x] code-reviewer ✅ on the harness record/replay fidelity ## Follow-up (separate PR — pipeline rule → human review) The tradein pytest gate currently runs **post-merge** (`deploy-tradein.yml`, deploy-gating). A true **pre-merge hard gate** needs a new `pull_request` job in `ci.yml` (extend the `changes` filter + a `backend-tests-tradein` job) + a Required check in branch protection. That edits pipeline rules → opened separately for human review. Refs #1966

lekss361 added 3 commits 2026-06-27 12:50:58 +00:00

feat(tradein/backtest): --dump-fixture + hermetic replay_fixture + prod-parity (anchor gate, drop MIN_CANDIDATES skip) (#1966 PR 3/3) 5a73bbeb8e

fix(tradein/backtest): order-based callable replay (platform-robust) + gzip fixture loading (#1966 PR 3/3) 1648da01c3

feat(tradein/backtest): frozen fixture + baseline + hermetic CI regression gate (#1966 PR 3/3)

CI / changes (pull_request) Successful in 7s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

f7d0bbd30e

Commits the gzipped 277-deal prod fixture (frozen _price_from_inputs inputs),
the frozen backtest_baseline.json (overall MAPE 18.63% / bias -1.2%; per-segment
бизнес -21.5% / элит -38.6% — the systemic expensive-segment underestimation,
now measured), and a hermetic pytest gate that replays the fixture through the
full pricing spine (ZERO DB) and asserts metrics == baseline within float tolerance.

Relative regression gate, not an absolute SLA (live coverage ~55% is data-blocked
per #1966). To change estimator behaviour: regenerate the baseline
(--from-fixture --update-baseline, no DB) and justify the per-segment deltas in the PR.
The fixture (gzipped prod inputs) re-extracts rarely; the per-change artifact is the
3 KB baseline. Large-file hook excludes the fixture (re-extracted only on ground-truth refresh).

Refs #1966