feat(sitefinder): rosreestr cadastral-value loader + 0.7x denoise of quarter price index #2068

Merged

lekss361 merged 1 commit from feat/rosreestr-cadastral-value-denoise into main

2026-06-28 17:02:13 +00:00

bot-backend commented

2026-06-28 16:46:45 +00:00

Collaborator

What / why

Adds a quarterly Rosreestr cadastral-value loader and uses it to denoise the
deals price index. The open dataset dataset_КАДАСТРСТОИМОСТЬ gives a per-cad-quarter
cadastral ₽/m² (cost_per_area). Registered deals priced below 0.7× the cadastral
median for their quarter are dropped from mv_quarter_price_per_m2 as noise
(data-entry junk / non-arm's-length deals that drag medians down).

Chain (consumers unchanged):
rosreestr_deals → mv_quarter_price_per_m2 (95) → mv_quarter_price_index (99a)
→ tradein FDW quarter_price_index.

Files

data/sql/178_schema_cadastral_value.sql — rosreestr_cadastral_value (+ _staging),
region-66-scoped, idempotent (CREATE TABLE/INDEX IF NOT EXISTS, BEGIN/COMMIT).
data/sql/178b_load_cadastral_value.py — psycopg v3 loader, mirrors
02b_load_via_tunnel.py: staging cur.copy() (UTF-8, ~ delimiter) → typed INSERT
filtering region_code='66' with regex-guarded numeric casts; idempotent skip-if-loaded.
data/sql/179_mv_quarter_price_cadastral_floor.sql — recreates both MVs
(DROP ... mv_quarter_price_per_m2 CASCADE also drops the dependent index MV). Adds a
cad_floor CTE + LEFT JOIN floor predicate to mv_quarter_price_per_m2;
mv_quarter_price_index recreated verbatim from 99a to restore the FDW contract.

Safety — no-op when cadastral table is empty

179 auto-applies on prod (via _schema_migrations) before any cadastral data is
loaded. The table exists (178) but is empty → cad_floor has zero rows → the
LEFT JOIN yields floor_ppm2 IS NULL for every quarter → the predicate
cf.floor_ppm2 IS NULL OR price_m2 >= cf.floor_ppm2 keeps all deals → the rebuilt
mv_quarter_price_per_m2 is byte-identical to the pre-floor (95) definition.
The floor only activates after an operator loads cadastral data and REFRESHes the MVs.

Dry-run (rolled back, against prod data — nothing mutated)

Used the existing experiment table cad_value_66_2026q1 (70 340 apartment rows, region 66)
as a stand-in cadastral source.

stage	source	66:% quarters	median ₽/m²
A — live MV	—	1972	66 788
B — dry, empty cad	(no-op)	1972	66 788
C — dry, floored	`cad_value_66_2026q1`	1967	67 350

No-op assertion (B): PASS — (quarter, deals_count, median) EXCEPT both directions =
0 / 0 (byte-identical to live for the 66:% scope).
Floor effect (C): PASS — 480 / 1972 quarters changed (475 lost deals, 5 dropped below
HAVING ≥ 3, 470 got a higher median). Samples:
66:41:0401048 252→199 deals, median 120 883→134 783;
66:41:0705005 326→279, median 115 294→122 388.

NOTE — data load is manual

The cadastral CSV is not auto-loaded. Deploy order:

Merge → migrations 178 + 179 auto-apply (MVs rebuilt, no behaviour change — cad table empty).
Operator downloads the quarter CSV to data/raw/ and runs
uv run python ../data/sql/178b_load_cadastral_value.py 2026Q1.
Operator REFRESH MATERIALIZED VIEW CONCURRENTLY mv_quarter_price_per_m2;
then ... mv_quarter_price_index; to activate the floor.

Deviation from spec

Migration numbers bumped 172/173 → 178/178b/179: 172–177 already exist on main
(spec assumed max=171). NN = next sequential per .claude/rules/sql.md.

🤖 Generated with Claude Code

## What / why Adds a quarterly **Rosreestr cadastral-value** loader and uses it to **denoise** the deals price index. The open dataset `dataset_КАДАСТРСТОИМОСТЬ` gives a per-cad-quarter cadastral ₽/m² (`cost_per_area`). Registered deals priced **below 0.7× the cadastral median** for their quarter are dropped from `mv_quarter_price_per_m2` as noise (data-entry junk / non-arm's-length deals that drag medians down). Chain (consumers unchanged): `rosreestr_deals` → `mv_quarter_price_per_m2` (95) → `mv_quarter_price_index` (99a) → tradein FDW `quarter_price_index`. ## Files - **`data/sql/178_schema_cadastral_value.sql`** — `rosreestr_cadastral_value` (+ `_staging`), region-66-scoped, idempotent (`CREATE TABLE/INDEX IF NOT EXISTS`, `BEGIN/COMMIT`). - **`data/sql/178b_load_cadastral_value.py`** — psycopg v3 loader, mirrors `02b_load_via_tunnel.py`: staging `cur.copy()` (UTF-8, `~` delimiter) → typed INSERT filtering `region_code='66'` with regex-guarded numeric casts; idempotent skip-if-loaded. - **`data/sql/179_mv_quarter_price_cadastral_floor.sql`** — recreates **both** MVs (`DROP ... mv_quarter_price_per_m2 CASCADE` also drops the dependent index MV). Adds a `cad_floor` CTE + `LEFT JOIN` floor predicate to `mv_quarter_price_per_m2`; `mv_quarter_price_index` recreated **verbatim** from 99a to restore the FDW contract. ## Safety — no-op when cadastral table is empty `179` auto-applies on prod (via `_schema_migrations`) **before** any cadastral data is loaded. The table exists (178) but is empty → `cad_floor` has **zero rows** → the `LEFT JOIN` yields `floor_ppm2 IS NULL` for **every** quarter → the predicate `cf.floor_ppm2 IS NULL OR price_m2 >= cf.floor_ppm2` keeps **all** deals → the rebuilt `mv_quarter_price_per_m2` is **byte-identical** to the pre-floor (95) definition. The floor only activates after an operator loads cadastral data and REFRESHes the MVs. ## Dry-run (rolled back, against prod data — nothing mutated) Used the existing experiment table `cad_value_66_2026q1` (70 340 apartment rows, region 66) as a stand-in cadastral source. | stage | source | 66:% quarters | median ₽/m² | |---|---|---|---| | A — live MV | — | 1972 | 66 788 | | B — dry, **empty** cad | (no-op) | 1972 | 66 788 | | C — dry, **floored** | `cad_value_66_2026q1` | 1967 | 67 350 | - **No-op assertion (B): PASS** — `(quarter, deals_count, median)` `EXCEPT` both directions = `0 / 0` (byte-identical to live for the `66:%` scope). - **Floor effect (C): PASS** — 480 / 1972 quarters changed (475 lost deals, 5 dropped below `HAVING ≥ 3`, 470 got a higher median). Samples: `66:41:0401048` 252→199 deals, median 120 883→134 783; `66:41:0705005` 326→279, median 115 294→122 388. ## NOTE — data load is manual The cadastral CSV is **not** auto-loaded. Deploy order: 1. Merge → migrations `178` + `179` auto-apply (MVs rebuilt, **no behaviour change** — cad table empty). 2. Operator downloads the quarter CSV to `data/raw/` and runs `uv run python ../data/sql/178b_load_cadastral_value.py 2026Q1`. 3. Operator `REFRESH MATERIALIZED VIEW CONCURRENTLY mv_quarter_price_per_m2;` then `... mv_quarter_price_index;` to activate the floor. ## Deviation from spec Migration numbers bumped **172/173 → 178/178b/179**: `172`–`177` already exist on `main` (spec assumed max=171). NN = next sequential per `.claude/rules/sql.md`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

bot-backend added 1 commit 2026-06-28 16:46:46 +00:00

feat(sitefinder): rosreestr cadastral-value loader + 0.7x denoise of quarter price index

CI / changes (pull_request) Successful in 8s

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Successful in 1m58s

Details

CI / backend-tests (pull_request) Successful in 14m30s

Details

4d063d6827

Quarterly Rosreestr dataset_КАДАСТРСТОИМОСТЬ (cadastral value) gives a
per-cad-quarter cadastral ₽/m². Registered deals priced below 0.7x that
quarter's cadastral median are dropped from mv_quarter_price_per_m2 as
noise (data-entry junk / non-arm's-length deals that drag medians down).

- 178_schema_cadastral_value.sql: rosreestr_cadastral_value (+ staging),
  region-66-scoped, idempotent.
- 178b_load_cadastral_value.py: psycopg v3 staging COPY -> typed region-66
  INSERT, idempotent skip-if-loaded. Run MANUALLY by operator after deploy.
- 179_mv_quarter_price_cadastral_floor.sql: recreates BOTH MVs (CASCADE
  drops the dependent index MV). Adds cad_floor CTE + LEFT JOIN guard to
  mv_quarter_price_per_m2; mv_quarter_price_index recreated verbatim.

Safe to auto-apply on prod before any cadastral data exists: empty
rosreestr_cadastral_value -> cad_floor has 0 rows -> floor_ppm2 IS NULL for
every quarter -> predicate keeps ALL deals -> MV byte-identical to the
pre-floor (95) definition. Proven by a rolled-back dry-run (66:% EXCEPT
both directions = 0). With the experiment cad source loaded the floor
moves 480 of 1972 quarters (median 66788 -> 67350).

NN bumped from spec's 172/173 to 178/178b/179 (172-177 already on main).