lekss361/gendesign

Fork 0

feat(tradein): СберИндекс time-adjust for frozen Rosreestr ДКП + per-dashboard filter fix (#794) #919

Merged

bot-reviewer merged 2 commits from feat/794-sber-time-adjust into main

2026-05-31 18:48:23 +00:00

lekss361 commented

2026-05-31 18:38:43 +00:00

Owner

Closes #794 (continues the data layer merged in #890/#887).

Context

#887 (PR #890) shipped the СберИндекс data layer (sber_index.py, table sber_price_index, monthly pull task + scheduler seed) but deliberately deferred the estimator wire (deliverable 3) and only half-did the asking benchmark (deliverable 4). It also shipped a filter bug. This PR closes the gaps and wires the coefficient live.

🔴 Bug fixed: per-dashboard filter

build_sber_route hardcoded SOURCE:SI + REALTY:2 for all three dashboards. Verified live against /api/sowa (2026-05-31):

dashboard	old filter (`SI`/`REALTY:2`)	correct filter
`real_estate_deals`	✅ 112 rows	—
`residential_real_estate_prices`	❌ 0 rows	`REAL_ESTATE_TYPE:1` + `REAL_ESTATE_NOVELTY:3` → 43 rows
`dinamika-tsen-obyavlenii`	❌ 0 rows	`SOURCE:DK` + `PRICE_TYPE:1` → 52 rows

So in prod the hedonic series (the preferred coefficient) and the asking benchmark were silently empty. Now each dashboard carries its own filter + segment field via a SberDashboard dataclass. All three pull the secondary-market (вторичка) ряд. Live-verified latest values: deals 97 973 ₽/м², hedonic 109 688, asking 96 052 (REF_AREA 66 = Свердловская область).

Deliverable 3 — estimator time-adjust wire (LIVE)

_fetch_dkp_corridor (#652 ДКП corridor) now re-bases each stale Rosreestr ДКП ppm² to the latest available index month:

adjusted_ppm² = deal_ppm² × idx[Свердловская обл., latest] / idx[Свердловская обл., deal_month]

Series preference: hedonic (residential_real_estate_prices) → deals (real_estate_deals) fallback.
Factor clamped to [0.7, 1.6]; graceful → 1.0 when sber_price_index is empty/missing (so it is a no-op until the fixed pull populates the table — zero blast radius today, prod table currently empty since the monthly scheduler hasn't fired).
_sber_time_factor is a pure, fully unit-tested function. Corridor return-dict shape unchanged; actual_deals display left untouched (shows recorded prices, not adjusted).

Deliverable 4 — asking benchmark reconciliation

After the dinamika (asking) upsert, logs our listings scrape-median asking ppm² vs the СберИндекс asking series + divergence %, gated to the ЕКБ region (REF_AREA 66) since our listings are EKB-only. Best-effort, log-only data-quality signal.

Tests

test_sber_index.py — per-dashboard route filter (locks the bug fix), decoder against a real captured fixture, period parse.
test_estimator_sber_time_adjust.py — _sber_time_factor edge cases (empty, latest, nearest-earlier, older-than-series, zero base, clamp hi/lo).
15 passed, ruff clean. Endpoint verified live (curl probe, values match baseline exactly).

Notes / follow-ups (non-blocking)

No new migration — table sber_price_index (092) is reused as-is.
Tech-debt (separate issues): in-process cache for the index series in the estimator hot path; SberDashboard.extra_filter immutability.

🤖 Generated with Claude Code

Closes #794 (continues the data layer merged in #890/#887). ## Context #887 (PR #890) shipped the СберИндекс data layer (`sber_index.py`, table `sber_price_index`, monthly pull task + scheduler seed) but deliberately deferred the estimator wire (deliverable 3) and only half-did the asking benchmark (deliverable 4). It also shipped a **filter bug**. This PR closes the gaps and wires the coefficient live. ## 🔴 Bug fixed: per-dashboard filter `build_sber_route` hardcoded `SOURCE:SI` + `REALTY:2` for **all three** dashboards. Verified live against `/api/sowa` (2026-05-31): | dashboard | old filter (`SI`/`REALTY:2`) | correct filter | |---|---|---| | `real_estate_deals` | ✅ 112 rows | — | | `residential_real_estate_prices` | ❌ **0 rows** | `REAL_ESTATE_TYPE:1` + `REAL_ESTATE_NOVELTY:3` → 43 rows | | `dinamika-tsen-obyavlenii` | ❌ **0 rows** | `SOURCE:DK` + `PRICE_TYPE:1` → 52 rows | So in prod the hedonic series (the *preferred* coefficient) and the asking benchmark were silently empty. Now each dashboard carries its own filter + segment field via a `SberDashboard` dataclass. All three pull the secondary-market (вторичка) ряд. Live-verified latest values: deals 97 973 ₽/м², hedonic 109 688, asking 96 052 (REF_AREA 66 = Свердловская область). ## Deliverable 3 — estimator time-adjust wire (LIVE) `_fetch_dkp_corridor` (#652 ДКП corridor) now re-bases each stale Rosreestr ДКП ppm² to the latest available index month: ``` adjusted_ppm² = deal_ppm² × idx[Свердловская обл., latest] / idx[Свердловская обл., deal_month] ``` - Series preference: hedonic (`residential_real_estate_prices`) → deals (`real_estate_deals`) fallback. - Factor clamped to **[0.7, 1.6]**; **graceful → 1.0** when `sber_price_index` is empty/missing (so it is a **no-op until the fixed pull populates the table** — zero blast radius today, prod table currently empty since the monthly scheduler hasn't fired). - `_sber_time_factor` is a pure, fully unit-tested function. Corridor return-dict shape unchanged; `actual_deals` display left untouched (shows recorded prices, not adjusted). ## Deliverable 4 — asking benchmark reconciliation After the dinamika (asking) upsert, logs our listings scrape-median asking ppm² vs the СберИндекс asking series + divergence %, gated to the ЕКБ region (REF_AREA 66) since our listings are EKB-only. Best-effort, log-only data-quality signal. ## Tests - `test_sber_index.py` — per-dashboard route filter (locks the bug fix), decoder against a **real captured fixture**, period parse. - `test_estimator_sber_time_adjust.py` — `_sber_time_factor` edge cases (empty, latest, nearest-earlier, older-than-series, zero base, clamp hi/lo). - 15 passed, ruff clean. Endpoint verified live (curl probe, values match baseline exactly). ## Notes / follow-ups (non-blocking) - No new migration — table `sber_price_index` (092) is reused as-is. - Tech-debt (separate issues): in-process cache for the index series in the estimator hot path; `SberDashboard.extra_filter` immutability. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

lekss361 added 2 commits 2026-05-31 18:38:44 +00:00

feat(tradein): СберИндекс time-adjust for frozen Rosreestr ДКП + per-dashboard filter fix (#794 ) 3485b977f1

fix(tradein): sber series fallback continues on error + benchmark gated to ЕКБ region (#794 review) 468bdb052b

lekss361 referenced this pull request

2026-05-31 18:39:00 +00:00

[research][tradein][backend] СберИндекс city-index → time-adjustment для замороженных Rosreestr deals (#724/#727) #794

lekss361 added the

status/review

label 2026-05-31 18:39:20 +00:00

bot-reviewer approved these changes 2026-05-31 18:48:15 +00:00

bot-reviewer left a comment

Collaborator

✅ APPROVE — корректно, inert-today, хорошо покрыто тестами. Мержу. Один actionable follow-up (backtest gate) заведу отдельным issue.

Verified (проверял сам, не по телу PR)

deals.deal_date существует — data/sql/002_core_tables.sql:107 (date NOT NULL, indexed). Добавленный SELECT price_per_m2, deal_date безопасен, регрессии corridor→None нет.
_sber_time_factor — pure, clamp [0.7, 1.6], graceful→1.0 на empty/zero-base/recent-deal; 8 edge-кейсов покрывают все ветки (empty, ==latest, >latest, normal, clamp hi/lo, nearest-earlier, older-than-series, zero-base).
_load_sber_index_series — CAST(:region AS text)/CAST(:dash AS text) (psycopg-v3 compliant, не :x::type); graceful {}; hedonic→deals preference.
No-op сегодня: прод sber_price_index пуст → factor 1.0 → нулевой blast radius; при заполнении clamp ограничивает эффект. actual_deals display не трогается, return-shape тот же.
Filter-fix: per-dashboard SberDashboard (slug/extra_filter/segment_field) — segment_field держит (city,period_month,dashboard) PK одно-значным. Locked parametrized-тестом. Decoder теперь читает имена полей из fields-cell (робастнее хардкод-порядка), провалидирован реальным фикстуром.
Benchmark reconciliation — best-effort, log-only, try/except graceful, gated REF_AREA 66. Не влияет на estimate.

🟡 Advisory — backtest gate (завожу follow-up issue)

#887/#890 осознанно отложил этот wire с пометкой «client-visible change requires a backtest gate». Этот PR его включает с unit-тестами, но без estimator-backtest. Сегодня это безопасно (table пуст → no-op), НО когда месячный scheduler заполнит sber_price_index, time-adjust активируется без валидации точности оценки. Митигировано clamp'ом и preference hedonic, но точность надо подтвердить через scripts/backtest_estimator.py ДО/сразу при активации. Завожу issue — не блокер (inert merge), но gate перед реальной активацией.

tech-debt (по плану автора, отдельные issue)

in-process cache для index-series в hot-path estimator (сейчас +1 query на _fetch_dkp_corridor).
SberDashboard.extra_filter immutability.

Squash-merge + delete branch. Closes #794.

✅ **APPROVE** — корректно, inert-today, хорошо покрыто тестами. Мержу. Один actionable follow-up (backtest gate) заведу отдельным issue. ## Verified (проверял сам, не по телу PR) - **`deals.deal_date` существует** — `data/sql/002_core_tables.sql:107` (`date NOT NULL`, indexed). Добавленный `SELECT price_per_m2, deal_date` безопасен, регрессии corridor→None нет. - **`_sber_time_factor`** — pure, clamp `[0.7, 1.6]`, graceful→1.0 на empty/zero-base/recent-deal; 8 edge-кейсов покрывают все ветки (empty, ==latest, >latest, normal, clamp hi/lo, nearest-earlier, older-than-series, zero-base). - **`_load_sber_index_series`** — `CAST(:region AS text)`/`CAST(:dash AS text)` (psycopg-v3 compliant, не `:x::type`); graceful `{}`; hedonic→deals preference. - **No-op сегодня**: прод `sber_price_index` пуст → factor 1.0 → нулевой blast radius; при заполнении clamp ограничивает эффект. `actual_deals` display не трогается, return-shape тот же. - **Filter-fix**: per-dashboard `SberDashboard` (slug/extra_filter/segment_field) — `segment_field` держит `(city,period_month,dashboard)` PK одно-значным. Locked parametrized-тестом. Decoder теперь читает имена полей из `fields`-cell (робастнее хардкод-порядка), провалидирован реальным фикстуром. - **Benchmark reconciliation** — best-effort, log-only, try/except graceful, gated REF_AREA 66. Не влияет на estimate. ## 🟡 Advisory — backtest gate (завожу follow-up issue) #887/#890 осознанно отложил этот wire с пометкой «client-visible change requires a **backtest gate**». Этот PR его включает с **unit-тестами, но без estimator-backtest**. Сегодня это безопасно (table пуст → no-op), НО когда месячный scheduler заполнит `sber_price_index`, time-adjust активируется **без валидации точности оценки**. Митигировано clamp'ом и preference hedonic, но точность надо подтвердить через `scripts/backtest_estimator.py` ДО/сразу при активации. Завожу issue — не блокер (inert merge), но gate перед реальной активацией. ## tech-debt (по плану автора, отдельные issue) - in-process cache для index-series в hot-path estimator (сейчас +1 query на `_fetch_dkp_corridor`). - `SberDashboard.extra_filter` immutability. Squash-merge + delete branch. `Closes #794`.

bot-reviewer merged commit 4999c877ae into main

2026-05-31 18:48:23 +00:00

bot-reviewer deleted branch feat/794-sber-time-adjust

2026-05-31 18:48:23 +00:00

bot-reviewer referenced this pull request from a commit

2026-05-31 18:48:25 +00:00

feat(tradein): СберИндекс time-adjust for frozen Rosreestr ДКП + per-dashboard filter fix (#794) (#919)

bot-reviewer referenced this pull request

2026-05-31 18:48:49 +00:00

[tradein][estimator] Backtest gate для СберИндекс time-adjust перед активацией (#794 follow-up) #920

lekss361 commented

2026-05-31 18:56:37 +00:00

Author

Owner

Pushed d2722df — fixes the 9 CI test regressions from the refactor (signature change broke pre-existing tests/test_sber_index.py + 2 dkp tests in test_estimator_imv_blend.py; also deduped a stray tests/services/test_sber_index.py). Full suite now green locally (1655 passed; the only 2 failures are the pre-existing app.main→psycopg2 import tests, unrelated, which pass in CI). app/ code unchanged in this commit — tests-only. CI re-running.

Pushed `d2722df` — fixes the 9 CI test regressions from the refactor (signature change broke pre-existing `tests/test_sber_index.py` + 2 dkp tests in `test_estimator_imv_blend.py`; also deduped a stray `tests/services/test_sber_index.py`). Full suite now green locally (1655 passed; the only 2 failures are the pre-existing `app.main`→`psycopg2` import tests, unrelated, which pass in CI). app/ code unchanged in this commit — tests-only. CI re-running.