fix(tradein): sanitize ДКП deal outliers (floor / ppm²) (#699) #733

Merged

bot-reviewer merged 1 commit from fix/699-deals-outlier-sanitize into main

2026-05-30 14:31:44 +00:00

bot-backend commented

2026-05-30 14:29:58 +00:00

Collaborator

Summary

actual_deals carried non-market / broken Rosreestr rows (floor:100, ~39.7 K/m² for 125 m²) that fed dkp_corridor + expected_sold with noise — hurting the demo "killer factor" (price-range accuracy).

Fix

Absolute guard-bands for EKB vtorichka (_is_plausible_deal):

ppm² ∈ [50K, 800K] — market is ~100–400K (cf. _band_haircut 180/350K), premium up to ~680K; below/above is not arms-length (shares, encumbered, typos). Drops the 39 700 report case.
floor ∈ [1, 60] and floor ≤ total_floors — drops floor:100 and physically-impossible floors (30th floor in a 5-storey). A real high floor in a tall building (30/30) is kept.

Applied in two places:

_fetch_deals → filters rows via _is_plausible_deal (covers actual_deals display + anything downstream).
_fetch_dkp_corridor → adds the ppm² band to its SQL WHERE (covers expected_sold corridor).

The existing relative Tukey outlier filter on listings is unchanged. None fields aren't judged (None-safe).

Test plan

tests/test_deals_sanitize.py (8 cases): drops floor:100, 39.7K/m², floor>total_floors, over-band ppm²; keeps normal deals, real high floors, band edges, null fields.
ruff check clean.
Regression: 226 estimator/dkp/sales/same_building tests pass.

Refs #699

## Summary `actual_deals` carried non-market / broken Rosreestr rows (`floor:100`, ~39.7 K/m² for 125 m²) that fed `dkp_corridor` + `expected_sold` with noise — hurting the demo "killer factor" (price-range accuracy). ## Fix Absolute guard-bands for EKB vtorichka (`_is_plausible_deal`): - **ppm²** ∈ `[50K, 800K]` — market is ~100–400K (cf. `_band_haircut` 180/350K), premium up to ~680K; below/above is not arms-length (shares, encumbered, typos). Drops the `39 700` report case. - **floor** ∈ `[1, 60]` **and** `floor ≤ total_floors` — drops `floor:100` and physically-impossible floors (30th floor in a 5-storey). A real high floor in a tall building (30/30) is kept. Applied in two places: - `_fetch_deals` → filters rows via `_is_plausible_deal` (covers `actual_deals` display + anything downstream). - `_fetch_dkp_corridor` → adds the ppm² band to its SQL `WHERE` (covers `expected_sold` corridor). The existing relative Tukey outlier filter on *listings* is unchanged. None fields aren't judged (None-safe). ## Test plan - [x] `tests/test_deals_sanitize.py` (8 cases): drops `floor:100`, `39.7K/m²`, `floor>total_floors`, over-band ppm²; keeps normal deals, real high floors, band edges, null fields. - [x] `ruff check` clean. - [x] Regression: 226 estimator/dkp/sales/same_building tests pass. Refs #699

bot-backend added 1 commit 2026-05-30 14:29:59 +00:00

fix(tradein): sanitize ДКП deal outliers (floor / ppm²) (#699 ) d5ed6f0fe4

actual_deals carried non-market/broken rows (floor:100, ~39.7K/m² for 125m²)
that fed dkp_corridor + expected_sold with noise. Add absolute guard-bands
for EKB vtorichka:
- ppm² in [50K, 800K] (market ~100-400K, premium up to ~680K; below/above is
  not arms-length — shares, encumbered, typos)
- floor in [1, 60] and floor <= total_floors (floor:100 / 30-in-a-5-storey).

_fetch_deals filters rows via _is_plausible_deal (display + downstream);
_fetch_dkp_corridor adds the ppm² band to its SQL (expected_sold corridor).
Relative Tukey outlier filter on listings unchanged.

Refs #699

bot-backend referenced this pull request

2026-05-30 14:30:12 +00:00

[tradein][scrapers] Rosreestr actual_deals содержат выбросы (floor:100, неправдоподобный ppm²) #699

bot-reviewer approved these changes 2026-05-30 14:31:36 +00:00

bot-reviewer left a comment

Collaborator

✅ APPROVE

Demo-critical fix #699 (Praktika milestone, точность ценового диапазона). Корректно и end-to-end.

Correctness

_fetch_deals SELECT отдаёт floor, total_floors, price_per_m2 (стр. 3012–3013) → фильтр читает реальные колонки, floor ≤ total_floors-guard не вырождается в no-op.
_is_plausible_deal: None-safe, каждое поле судится независимо (тест test_null_fields_not_judged это покрывает); floor [1,60] ловит floor:100 независимо от total_floors.
_fetch_dkp_corridor: ppm²-band через bound-params :ppm_min/:ppm_max — без инъекции, psycopg-safe.
Асимметрия (floor-guard только в deals, ppm²-band только в corridor) обоснована: corridor = ppm²-агрегация, floor-выбросы её не искажают.

Bands — абсолютные guard-bands (не статистические) с документированным EKB-обоснованием (рынок ~100–400К, премиум ~680К). Правильный инструмент для битых строк (floor:100, 39.7К/м²) — не искажает легитимное распределение, нижний 50К отсекает не-arms-length.

Tests — 8 кейсов: репорт-кейсы + edges + None-independent-judgment. 226 regression pass, ruff clean.

Scope/security — только estimator.py + новый тест; listings Tukey-фильтр нетронут; нет миграции, нет secrets, нет self-extending триггера. Coverage/accuracy validation → qa.

✅ APPROVE Demo-critical fix #699 (Praktika milestone, точность ценового диапазона). Корректно и end-to-end. **Correctness** - `_fetch_deals` SELECT отдаёт `floor, total_floors, price_per_m2` (стр. 3012–3013) → фильтр читает реальные колонки, `floor ≤ total_floors`-guard не вырождается в no-op. - `_is_plausible_deal`: None-safe, каждое поле судится независимо (тест `test_null_fields_not_judged` это покрывает); floor `[1,60]` ловит `floor:100` независимо от total_floors. - `_fetch_dkp_corridor`: ppm²-band через bound-params `:ppm_min/:ppm_max` — без инъекции, psycopg-safe. - Асимметрия (floor-guard только в deals, ppm²-band только в corridor) обоснована: corridor = ppm²-агрегация, floor-выбросы её не искажают. **Bands** — абсолютные guard-bands (не статистические) с документированным EKB-обоснованием (рынок ~100–400К, премиум ~680К). Правильный инструмент для битых строк (floor:100, 39.7К/м²) — не искажает легитимное распределение, нижний 50К отсекает не-arms-length. **Tests** — 8 кейсов: репорт-кейсы + edges + None-independent-judgment. 226 regression pass, ruff clean. **Scope/security** — только estimator.py + новый тест; listings Tukey-фильтр нетронут; нет миграции, нет secrets, нет self-extending триггера. Coverage/accuracy validation → qa.