3) #1983

Merged

lekss361 merged 3 commits from feat/tradein-backtest-full-spine into main

2026-06-27 11:21:58 +00:00

lekss361 commented

2026-06-27 11:15:47 +00:00

Owner

Summary

#1966 PR 2/3. Upgrades the read-only backtest harness from the asking-median core to the full deterministic spine via _price_from_inputs (the #1976 extraction), and adds range-coverage, confidence-calibration, and per-price-segment MAPE (on top of per-rooms).

Real prod numbers (full engine, freshest deals since 2025-10-01, n=371 matched / 400)

expected_sold vs actual rosreestr ДКП sold:

Overall: MAPE 18.4%, bias −1.2% (well-centered; ≈ asking-core corrected 19.6%, spine slightly better).
Per-price-segment — the honest picture (confirms strategic audit 0627):

segment n bias MAPE

эконом 170 +16.4% 17.8%

комфорт 135 −8.1% 16.5%

бизнес 56 −22.7% 22.9%

элит 9 −37.9% 37.9%

премиум 1 −59% (noise) —

→ the engine overvalues эконом and systematically UNDERvalues the expensive segments (бизнес/элит) — exactly where the trade-in money is. Now measured, not hand-waved.
Per-rooms: студия +23.7% / 1к −7% / 2к −8.3% / 3к +5% (MAPE 12.4%, best) / 4+ +10.8%.
Range coverage 56.6% overall — the predicted range is miscalibrated (too narrow for honest coverage); R2 risk now quantified. Sharpness (median rel width) 0.47.
Calibration inconclusive in this run: confidence ≈ always "low" (backtest doesn't resolve target_house_id → same-building Tier-A anchor never fires → high/medium buckets n=1/n=3). Documented caveat.
Headline ask vs sold spread +19.2% (vs +31% asking-core — the spine pulls the headline toward sold).

segment	n	bias	MAPE
эконом	170	+16.4%	17.8%
комфорт	135	−8.1%	16.5%
бизнес	56	−22.7%	22.9%
элит	9	−37.9%	37.9%
премиум	1	−59% (noise)	—

Caveat (standing): CURRENT listings vs PAST deals → time-mismatched → this is a regression baseline, not an absolute SLA. Network valuation layers (on-demand Avito-IMV / Yandex / Cian) excluded for hermeticity.

What's in

_select_analogs_full: byte-faithful replication of estimate_quality's analog tier ladder (Tier 0 cohort → A → wide → widearea) via _fetch_analogs (full lot dicts).
_price_from_inputs composition with injected ratio_resolver / quarter lookups + a synthetic GeocodeResult (client-coords fast path).
Pure, unit-tested metrics: _range_coverage, _calibration_metrics, _segment_metrics, _sharpness (55 tests in the module; full suite 2488 passed, ruff clean).
CLI --engine {full,asking-core} (default full); --json. PRICE_SEGMENTS_PPM2 = documented tunable EKB bands.

Unblocks PR 3/3 (wire these metrics as a CI regression gate on a frozen backtest_baseline.json).

Part of #1966 (PR 2/3). Refs #1966.

## Summary #1966 PR 2/3. Upgrades the read-only backtest harness from the asking-median **core** to the **full deterministic spine** via `_price_from_inputs` (the #1976 extraction), and adds **range-coverage**, **confidence-calibration**, and **per-price-segment** MAPE (on top of per-rooms). ## Real prod numbers (full engine, freshest deals since 2025-10-01, n=371 matched / 400) `expected_sold` vs actual rosreestr ДКП sold: - **Overall: MAPE 18.4%, bias −1.2%** (well-centered; ≈ asking-core corrected 19.6%, spine slightly better). - **Per-price-segment — the honest picture (confirms strategic audit 0627):** | segment | n | bias | MAPE | |---|---|---|---| | эконом | 170 | **+16.4%** | 17.8% | | комфорт | 135 | −8.1% | 16.5% | | бизнес | 56 | **−22.7%** | 22.9% | | элит | 9 | **−37.9%** | 37.9% | | премиум | 1 | −59% (noise) | — | → the engine **overvalues эконом and systematically UNDERvalues the expensive segments** (бизнес/элит) — exactly where the trade-in money is. Now measured, not hand-waved. - Per-rooms: студия +23.7% / 1к −7% / 2к −8.3% / **3к +5% (MAPE 12.4%, best)** / 4+ +10.8%. - **Range coverage 56.6%** overall — the predicted range is miscalibrated (too narrow for honest coverage); R2 risk now quantified. Sharpness (median rel width) 0.47. - **Calibration inconclusive in this run**: confidence ≈ always "low" (backtest doesn't resolve `target_house_id` → same-building Tier-A anchor never fires → high/medium buckets n=1/n=3). Documented caveat. - Headline ask vs sold spread **+19.2%** (vs +31% asking-core — the spine pulls the headline toward sold). **Caveat (standing):** CURRENT listings vs PAST deals → time-mismatched → this is a **regression baseline, not an absolute SLA**. Network valuation layers (on-demand Avito-IMV / Yandex / Cian) excluded for hermeticity. ## What's in - `_select_analogs_full`: byte-faithful replication of `estimate_quality`'s analog tier ladder (Tier 0 cohort → A → wide → widearea) via `_fetch_analogs` (full lot dicts). - `_price_from_inputs` composition with injected `ratio_resolver` / quarter lookups + a synthetic `GeocodeResult` (client-coords fast path). - Pure, unit-tested metrics: `_range_coverage`, `_calibration_metrics`, `_segment_metrics`, `_sharpness` (55 tests in the module; full suite **2488 passed**, ruff clean). - CLI `--engine {full,asking-core}` (default `full`); `--json`. `PRICE_SEGMENTS_PPM2` = documented tunable EKB bands. Unblocks **PR 3/3** (wire these metrics as a CI regression gate on a frozen `backtest_baseline.json`). Part of #1966 (PR 2/3). Refs #1966.

lekss361 added 3 commits 2026-06-27 11:15:48 +00:00