feat(tradein/estimator): calibrated 80% prediction interval for expected_sold range (#1966) #2001

Merged

lekss361 merged 1 commit from feat/tradein-calibrated-expected-sold-pi into main

2026-06-27 14:01:25 +00:00

lekss361 commented

2026-06-27 14:00:54 +00:00

Owner

Summary

Honest uncertainty for the trade-in estimate range (option B from the #1966 accuracy investigation). The old expected_sold_range was the analog asking-IQR × ratio — a ~50% interval mislabeled as the estimate's range, so only 55% of actual sold prices landed inside it. Replaced with a calibrated ~80% prediction interval around the expected-sold point.

Empirical: over 2366 prod deals, actual_sold / expected_sold has p10 = 0.649, p90 = 1.392.
New range = expected_sold_price × [0.649, 1.392] → a genuine 80% PI.
Gated behind estimate_calibrated_pi_enabled (default ON); estimate_pi_low_mult / estimate_pi_high_mult are tunable settings.

Measured effect (frozen backtest gate, regenerated hermetically)

metric	before	after
range_coverage.overall	55.23% (153/277)	80.14% (222/277)
sharpness median_rel_width	0.464	0.743 (= 1.392 − 0.649)
expected_sold point (mape/bias, per_rooms, per_segment)	—	unchanged
headline (asking IQR) range	—	unchanged

The point estimate is untouched — this is purely honest uncertainty quantification. The PI is wider (the honest cost of true 80% coverage vs the old mislabeled ~50% band) and asymmetric upward (luxury units sell above asking).

Why this and not point-recalibration

The investigation proved the per-segment point bias (элит −34%, эконом +17%) is unidentifiable from es (R²=0.16; a global g(es) recalibration worsens the tails — verified). The neighborhood signals (asking pool, micro-comps, same-address ДКП) all underprice elit because the premium lives in unobserved building quality — and the building-quality columns (house_class) are 0% populated, so a feature fix is gated on an enrichment backfill. The honest near-term move is calibrated uncertainty + (follow-up PR-2) a manual-review flag for high-value/low-confidence estimates.

Test plan

gate tests/test_backtest_regression_gate.py green (replay == regenerated baseline)
full tradein suite 2502 passed / 2 skipped (CI-equivalent)
4 estimator tests updated to the calibrated band + a new low ≤ point ≤ high invariant (strengthened, not weakened) — reviewer-verified legitimate
code-reviewer ✅ (point untouched, headline untouched, no test weakening)
band cross-checked on the live 2366-deal sample → 80.0% coverage

Follow-up

PR-2: manual-review flag (estimator → API → DB → frontend) for high-value / low-confidence units where the band can't honestly cover (elit median sold = 1.51× es, above the 1.392 upper bound).

Refs #1966

## Summary Honest uncertainty for the trade-in estimate range (option **B** from the #1966 accuracy investigation). The old `expected_sold_range` was the analog **asking-IQR × ratio** — a ~50% interval mislabeled as the estimate's range, so only **55%** of actual sold prices landed inside it. Replaced with a **calibrated ~80% prediction interval** around the expected-sold point. - Empirical: over **2366 prod deals**, `actual_sold / expected_sold` has **p10 = 0.649, p90 = 1.392**. - New range = `expected_sold_price × [0.649, 1.392]` → a genuine 80% PI. - Gated behind `estimate_calibrated_pi_enabled` (default ON); `estimate_pi_low_mult` / `estimate_pi_high_mult` are tunable settings. ## Measured effect (frozen backtest gate, regenerated hermetically) | metric | before | after | |---|---|---| | range_coverage.overall | **55.23%** (153/277) | **80.14%** (222/277) | | sharpness median_rel_width | 0.464 | 0.743 (= 1.392 − 0.649) | | **expected_sold point** (mape/bias, per_rooms, per_segment) | — | **unchanged** | | headline (asking IQR) range | — | **unchanged** | The **point estimate is untouched** — this is purely honest uncertainty quantification. The PI is wider (the honest cost of true 80% coverage vs the old mislabeled ~50% band) and asymmetric upward (luxury units sell above asking). ## Why this and not point-recalibration The investigation proved the per-segment point bias (элит −34%, эконом +17%) is **unidentifiable from `es`** (R²=0.16; a global `g(es)` recalibration *worsens* the tails — verified). The neighborhood signals (asking pool, micro-comps, same-address ДКП) **all** underprice elit because the premium lives in unobserved building quality — and the building-quality columns (`house_class`) are **0% populated**, so a feature fix is gated on an enrichment backfill. The honest near-term move is calibrated uncertainty + (follow-up PR-2) a manual-review flag for high-value/low-confidence estimates. ## Test plan - [x] gate `tests/test_backtest_regression_gate.py` green (replay == regenerated baseline) - [x] full tradein suite **2502 passed / 2 skipped** (CI-equivalent) - [x] 4 estimator tests updated to the calibrated band + a new `low ≤ point ≤ high` invariant (strengthened, not weakened) — reviewer-verified legitimate - [x] code-reviewer ✅ (point untouched, headline untouched, no test weakening) - [x] band cross-checked on the live 2366-deal sample → 80.0% coverage ## Follow-up - **PR-2**: manual-review flag (estimator → API → DB → frontend) for high-value / low-confidence units where the band can't honestly cover (elit median sold = 1.51× es, above the 1.392 upper bound). Refs #1966

lekss361 added 1 commit 2026-06-27 14:00:54 +00:00

feat(tradein/estimator): calibrated 80% prediction interval for expected_sold range (#1966 )

CI / changes (pull_request) Successful in 6s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

ed4335b932

The expected_sold_range_low/high were derived from the analog asking-IQR
(× asking→sold ratio): only ~55% of actual sold prices fell inside the stated
range — a 50%-ish interval mislabeled as the estimate's range. Replace with a
calibrated band around the expected_sold POINT, gated behind
estimate_calibrated_pi_enabled (default ON):

  range_low  = expected_sold_price × 0.649  (empirical p10 of sold/expected_sold)
  range_high = expected_sold_price × 1.392  (empirical p90, n=2366 prod deals)

This is a genuine ~80% prediction interval (verified 80.0% coverage on the
2366-deal sample). Headline asking range_low/high (market-asking IQR) untouched.

Frozen backtest baseline regenerated: range_coverage.overall 55.23 → 80.14
(n_covered 153 → 222); expected_sold POINT mape/bias and headline unchanged
(only the band moved). Tests asserting the old asking-IQR × ratio band updated
to the calibrated band (documented behavior change).