feat(obj-2): mapping backfill via pre-materialized temp tables (migration 117) #327

Merged

lekss361 merged 1 commit from feat/117-obj2-mapping-backfill-fast into main

2026-05-17 20:34:42 +00:00

lekss361 commented

2026-05-17 20:29:30 +00:00

Owner

Summary

Fix coverage regression в Site Finder после deploy OBJ-3 PR #324.

Problem

User report: на участке 66:41:0204016:10 velocity показывает 0, «срок продажи 1092 мес» overflow. Coverage = 3/20 ЖК (15%) для competitors блока. 17 ЖК Железнодорожного района без objective_complex_mapping запись → LEFT JOIN отдаёт NULL → aggregates broken.

Live verification (POST /parcels/.../analyze)

3 ЖК с правильными данными (acceptance criterion соблюдается):

ЖК	₽/м²	площадь	sold	avail
Квартал "Траектория"	143461	57.0	279	1356
Астон. Движение	145003	35.8	569	1791
Жилой Комплекс "Раута"	139038	44.5	1613	2321

Цены ∈ [110-220k] ✅. Но 17/20 без данных.

Fix

Migration 117 делает фактический mapping INSERT который был skip'нут в hotfix #326:

Pre-materialize unmapped DOM.РФ ЕКБ objects → _u_domrf (~1056 rows)
Pre-materialize unmapped Objective projects → _u_obj (~178 rows)
CROSS JOIN LATERAL top-3 per obj_id, leveraging GIST trgm index (создан в #326)
Composite score = name_sim*0.6 + dev_match*0.25 + district_match*0.15
INSERT auto-accepted (composite ≥ 0.75) с ON CONFLICT DO NOTHING
REFRESH mv_layout_velocity CONCURRENTLY

Why fast

Original 116 CTE recomputed unmapped lists для каждой строки → 89M cost timeout. Pre-materialized temp tables вычисляются один раз, далее GIST trgm index делает per-row LIMIT 1 dirt-cheap → estimated <30s runtime.

Expected outcome

objective_complex_mapping count: 142 → ~200 (+50-80)
Site Finder coverage Железнодорожный: 15% → ~40-60%
Velocity больше не показывает 0 для mixed districts

Acceptance

После deploy + REFRESH MV:

curl -X POST https://gendsgn.ru/api/v1/parcels/66:41:0204016:10/analyze \
  -H "Content-Type: application/json" -d '{}' \
  | jq '[.competitors[] | select(.avg_price_per_m2_rub != null)] | length'
# Ожидается: 8+ (вместо текущих 3)

Part of issue #307 OBJ-2 (deferred INSERT from migration 116 hotfix #326).

## Summary Fix coverage regression в Site Finder после deploy OBJ-3 PR #324. ## Problem User report: на участке `66:41:0204016:10` velocity показывает 0, «срок продажи 1092 мес» overflow. Coverage = **3/20 ЖК (15%)** для competitors блока. 17 ЖК Железнодорожного района без `objective_complex_mapping` запись → LEFT JOIN отдаёт NULL → aggregates broken. ## Live verification (POST /parcels/.../analyze) 3 ЖК с правильными данными (acceptance criterion соблюдается): | ЖК | ₽/м² | площадь | sold | avail | |---|---|---|---|---| | Квартал "Траектория" | **143461** | 57.0 | 279 | 1356 | | Астон. Движение | **145003** | 35.8 | 569 | 1791 | | Жилой Комплекс "Раута" | **139038** | 44.5 | 1613 | 2321 | Цены ∈ [110-220k] ✅. Но 17/20 без данных. ## Fix Migration **117** делает фактический mapping INSERT который был skip'нут в hotfix #326: 1. Pre-materialize unmapped DOM.РФ ЕКБ objects → `_u_domrf` (~1056 rows) 2. Pre-materialize unmapped Objective projects → `_u_obj` (~178 rows) 3. CROSS JOIN LATERAL top-3 per obj_id, leveraging GIST trgm index (создан в #326) 4. Composite score = `name_sim*0.6 + dev_match*0.25 + district_match*0.15` 5. INSERT auto-accepted (composite ≥ 0.75) с `ON CONFLICT DO NOTHING` 6. REFRESH `mv_layout_velocity` CONCURRENTLY ## Why fast Original 116 CTE recomputed unmapped lists для каждой строки → **89M cost** timeout. Pre-materialized temp tables вычисляются **один раз**, далее GIST trgm index делает per-row LIMIT 1 dirt-cheap → estimated **<30s** runtime. ## Expected outcome - `objective_complex_mapping` count: 142 → **~200** (+50-80) - Site Finder coverage Железнодорожный: 15% → **~40-60%** - Velocity больше не показывает 0 для mixed districts ## Acceptance После deploy + REFRESH MV: ```bash curl -X POST https://gendsgn.ru/api/v1/parcels/66:41:0204016:10/analyze \ -H "Content-Type: application/json" -d '{}' \ | jq '[.competitors[] | select(.avg_price_per_m2_rub != null)] | length' # Ожидается: 8+ (вместо текущих 3) ``` Part of issue #307 OBJ-2 (deferred INSERT from migration 116 hotfix #326).

lekss361 added 1 commit 2026-05-17 20:29:30 +00:00

feat(obj-2): mapping backfill via pre-materialized temp tables (117) 5f50a7ec96

Continues from migration 116 (trimmed to GIST index only after timeout).

Algorithm:
1. Materialize unmapped DOM.РФ ЕКБ objects → _u_domrf (~1056 rows)
2. Materialize unmapped Objective projects → _u_obj (~178 rows)
3. CROSS JOIN LATERAL with top-3 per obj_id, leveraging GIST trgm index from 116
4. Composite score = name_sim*0.6 + dev_match*0.25 + district_match*0.15
5. INSERT auto-accepted (composite >= 0.75) with ON CONFLICT DO NOTHING

Why pre-materialize: original CTE recomputed unmapped lists for every row,
costing 89M and timing out. Temp tables compute once + GIST index = <30s.

Expected: +50-80 mappings, total ~200. Coverage Site Finder competitors
улучшится с 15% до ~40-60% (особенно для районов с Объективными devs).

lekss361 commented

2026-05-17 20:34:36 +00:00

Author

Owner

Deep Code Review — PR #327

Verdict: APPROVE

Single-file SQL migration, +91/-0. Pre-materialization strategy is sound and actually addresses the 89M-cost timeout (not cosmetic).

Cardinality / cost analysis

The original 116 timeout came from per-row recomputation of NOT EXISTS (mapping lookup) inside CROSS JOIN LATERAL over the full domrf_kn_objects × objective_lots product. Three independent optimizations stack here:

_u_domrf (~1056 rows) — NOT EXISTS evaluated once during materialization.
_u_obj (~178 distinct rows) — NOT IN (... mapping ...) evaluated once, plus DISTINCT collapses duplicate (project_name, developer, district) tuples from objective_lots.
LATERAL scans _u_obj only (~178 rows) per outer _u_domrf row → 1056 × 178 = 188k similarity() calls total. Each is μs-cheap → well under 30s without any index help.

Note on GIST trgm index claim

PR description says LATERAL "leverages GIST trgm index from migration 116". Strictly that index is on objective_lots.project_name, not on the temp table _u_obj — temp tables don't inherit source indexes. The real performance win is the cardinality collapse, not the index. The migration is still fast (188k similarity calls is trivial), so this is a documentation nit, not a correctness issue.

Correctness checklist

Composite weights sum to 1.0 (0.6 + 0.25 + 0.15) ✓
Threshold composite >= 0.75 matches #323 ✓
ROW_NUMBER() OVER (PARTITION BY obj_id ORDER BY composite DESC, name_sim DESC) with tiebreaker — deterministic ✓
ON CONFLICT (objective_complex_name, objective_group) DO NOTHING — matches verified UNIQUE constraint, idempotent on re-apply ✓
COALESCE for NULL dev_name / district_name ✓
dev_match guards against dev_norm <> '' (avoids false-positive when both sides normalize to empty) ✓

Transaction boundary

BEGIN; ... CREATE TEMP TABLE ... ON COMMIT DROP ... INSERT ... COMMIT; ✓
REFRESH MATERIALIZED VIEW CONCURRENTLY mv_layout_velocity; outside BEGIN/COMMIT ✓ — required because CONCURRENTLY cannot run in an explicit transaction block. mv_layout_velocity_pk unique index exists (verified in #323).

Idempotency / retry safety

All three temp tables: ON COMMIT DROP — auto-cleanup ✓
INSERT under ON CONFLICT DO NOTHING — partial reruns safe ✓
If REFRESH MV step fails after COMMIT: rerun won't double-insert (DO NOTHING), and REFRESH itself is idempotent.

Minor nit (low severity, not blocking)

Line 17, 31: regexp_replace(lower(...), '[^а-яa-z0-9]', '', 'g') — the Cyrillic char class а-я (U+0430-U+044F) does not include ё (U+0451). A developer name containing ё will have that letter stripped. Comparison stays symmetric (both sides apply same regex), so dev_match still works correctly — just a known minor coverage edge for any developer with ё in the name. No fix needed for this PR.

Risk

Low risk: INSERT-only, idempotent, no DROP/TRUNCATE, no schema changes.
Reversibility: trivial — DELETE FROM objective_complex_mapping WHERE match_method='multi_feature_v1' AND note LIKE '%117_fast%' if rollback needed.
Recommended merge window: anytime.

Merging.

## Deep Code Review — PR #327 ### Verdict: APPROVE Single-file SQL migration, +91/-0. Pre-materialization strategy is sound and actually addresses the 89M-cost timeout (not cosmetic). ### Cardinality / cost analysis The original 116 timeout came from per-row recomputation of `NOT EXISTS (mapping lookup)` inside `CROSS JOIN LATERAL` over the full `domrf_kn_objects × objective_lots` product. Three independent optimizations stack here: 1. **`_u_domrf`** (~1056 rows) — `NOT EXISTS` evaluated once during materialization. 2. **`_u_obj`** (~178 distinct rows) — `NOT IN (... mapping ...)` evaluated once, plus `DISTINCT` collapses duplicate (project_name, developer, district) tuples from `objective_lots`. 3. **LATERAL** scans `_u_obj` only (~178 rows) per outer `_u_domrf` row → 1056 × 178 = **188k similarity() calls total**. Each is μs-cheap → well under 30s without any index help. ### Note on GIST trgm index claim PR description says LATERAL "leverages GIST trgm index from migration 116". Strictly that index is on `objective_lots.project_name`, not on the temp table `_u_obj` — temp tables don't inherit source indexes. The real performance win is the cardinality collapse, not the index. The migration is still fast (188k similarity calls is trivial), so this is a documentation nit, not a correctness issue. ### Correctness checklist - Composite weights sum to 1.0 (0.6 + 0.25 + 0.15) ✓ - Threshold `composite >= 0.75` matches #323 ✓ - `ROW_NUMBER() OVER (PARTITION BY obj_id ORDER BY composite DESC, name_sim DESC)` with tiebreaker — deterministic ✓ - `ON CONFLICT (objective_complex_name, objective_group) DO NOTHING` — matches verified UNIQUE constraint, idempotent on re-apply ✓ - `COALESCE` for NULL dev_name / district_name ✓ - `dev_match` guards against `dev_norm <> ''` (avoids false-positive when both sides normalize to empty) ✓ ### Transaction boundary - `BEGIN; ... CREATE TEMP TABLE ... ON COMMIT DROP ... INSERT ... COMMIT;` ✓ - `REFRESH MATERIALIZED VIEW CONCURRENTLY mv_layout_velocity;` **outside** BEGIN/COMMIT ✓ — required because CONCURRENTLY cannot run in an explicit transaction block. `mv_layout_velocity_pk` unique index exists (verified in #323). ### Idempotency / retry safety - All three temp tables: `ON COMMIT DROP` — auto-cleanup ✓ - INSERT under `ON CONFLICT DO NOTHING` — partial reruns safe ✓ - If REFRESH MV step fails after COMMIT: rerun won't double-insert (DO NOTHING), and REFRESH itself is idempotent. ### Minor nit (low severity, not blocking) Line 17, 31: `regexp_replace(lower(...), '[^а-яa-z0-9]', '', 'g')` — the Cyrillic char class `а-я` (U+0430-U+044F) does **not** include `ё` (U+0451). A developer name containing `ё` will have that letter stripped. Comparison stays symmetric (both sides apply same regex), so dev_match still works correctly — just a known minor coverage edge for any developer with `ё` in the name. No fix needed for this PR. ### Risk - Low risk: INSERT-only, idempotent, no DROP/TRUNCATE, no schema changes. - Reversibility: trivial — `DELETE FROM objective_complex_mapping WHERE match_method='multi_feature_v1' AND note LIKE '%117_fast%'` if rollback needed. - Recommended merge window: anytime. Merging.