27 commits

Author	SHA1	Message	Date
lekss361	e673811a20	feat(parcels): layout signature schemas + extractor (#113 PR A) (#193)	2026-05-16 07:52:34 +00:00
lekss361	4deb1b0284	feat(parcels): connection-points endpoint (Forgejo #115 Phase 1 backend) (#190)	2026-05-16 05:58:27 +00:00
lekss361	e561df1b55	feat(parcels): competitors endpoint (Forgejo #112 Phase 1 backend) (#191)	2026-05-15 22:21:05 +00:00
lekss361	d7fbaa0528	feat(velocity): per-room-bucket breakdown in VelocityResult (#163) ... * fix(velocity): :window_interval::interval cast syntax — same bug as PR #152 SQLAlchemy text() parser confuses ':window_interval' (named param) followed by '::interval' (cast operator) — exactly как было в PR #152 с :weights::jsonb. Result: psycopg видит literal ':window_interval' в SQL → syntax error → exception caught by velocity catch → return None → UI shows null. ## Fix ':window_interval::interval' → 'CAST(:window_interval AS interval)' (2 мест: sales_rows query + _get_ekb_median percentile). Pre-push code-reviewer должен был catch — добавим в feedback что для каждого text() SQL grep ':[a-z]*::' before push. Refs: PR #158 deploy verify, e2e velocity = null root cause * feat(velocity): add per-room-bucket breakdown to VelocityResult Add third SQL query (bucket_rows) aggregating deals_total_count by room_bucket ('студия','1','2','3','4+') across mapped competitors. New fields: - VelocityResult.by_room_bucket: aggregate {units, sqm, complexes_count} - sample_competitors[].by_room_bucket: {bucket: units} Bucket query wrapped в SAVEPOINT — failure degrades gracefully к empty dict без abort outer tx. CAST(:window_interval AS interval) pattern per PR #160 lesson. Test coverage: 3 new tests (aggregation, empty fallback, sample entries). Closes part of #161 --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 10:09:03 +03:00
lekss361	46584d7c75	fix(velocity): :window_interval::interval cast syntax — same bug as PR #152 (#160) ... SQLAlchemy text() parser confuses ':window_interval' (named param) followed by '::interval' (cast operator) — exactly как было в PR #152 с :weights::jsonb. Result: psycopg видит literal ':window_interval' в SQL → syntax error → exception caught by velocity catch → return None → UI shows null. ## Fix ':window_interval::interval' → 'CAST(:window_interval AS interval)' (2 мест: sales_rows query + _get_ekb_median percentile). Pre-push code-reviewer должен был catch — добавим в feedback что для каждого text() SQL grep ':[a-z]*::' before push. Refs: PR #158 deploy verify, e2e velocity = null root cause Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 09:46:01 +03:00
lekss361	8fa1d005fd	feat(velocity): migrate D2 sales source → objective_corpus_room_month (#158) ... * feat(velocity): migrate D2 sales source from stale sale_graph to objective_corpus_room_month Closes #156 * fix(velocity): cross-stack contract + NULL safety per pre-push review 3 pre-push code-reviewer findings fixed: 1. TS union extended ('objective'|'sale_graph'|'rosreestr_fallback') + UI conditions handle both objective и sale_graph как valid sources. 2. COALESCE(deals_total_vol_m2, deals_total_count * 45.0) — NULL safety for DKP-only rows (vol_m2 nullable, count > 0). 3. room_bucket parking filter — verified false positive (все 5 buckets apartments: студия/1/2/3/4+). Refs: PR #157 pre-push code-reviewer * fix(types): add 'objective' to velocity_source union (cherry-pick miss) Cherry-pick от закрытого PR #157 потерял этот файл. Frontend type-check fails с 'no overlap' error. Restoring. Refs: PR #158 frontend CI failure --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 09:34:00 +03:00
lekss361	c39b53cf53	fix(site-finder): cascade InFailedSqlTransaction — velocity internal rollback (#155) ... * fix(site-finder): db.rollback() в velocity catches + SAVEPOINT zoning/success_rec velocity.py 3 internal try/except возвращали None БЕЗ db.rollback(). PR #154 SAVEPOINT не помог — exception не propagates из velocity. Postgres tx aborted внутри, cascade InFailedSqlTransaction на следующих queries. Fix: - velocity.py: db.rollback() в 3 catch блоках - parcels.py: SAVEPOINT обёртки для zoning + success_recommendation Refs: user reports 2026-05-15 cascade 500 * fix(velocity): SAVEPOINT per query (not db.rollback) — correct SQLAlchemy 2.0 pattern Per PR #155 bot review — мой db.rollback() в catches был НЕПРАВИЛЬНЫЙ. Session.rollback() = ENTIRE outer tx (не savepoint). Outer SAVEPOINT context становится orphaned → InvalidRequestError на __exit__. Correct fix: wrap каждый db.execute в свой with db.begin_nested(): - Failure → __exit__ propagates exception → SAVEPOINT rolls back - Outer tx остаётся clean - velocity returns None gracefully → caller продолжает Все 3 SQL queries в velocity.py теперь SAVEPOINT-wrapped: - compute_velocity: competitor_query - compute_velocity: sales_rows query - _get_ekb_median: median query db.rollback() removed из всех 3 catches. Refs: PR #155 bot review --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 08:52:38 +03:00
lekss361	af39e95f83	fix(site-finder): SQLAlchemy syntax error :weights::jsonb → CAST(:weights AS jsonb) (#152) ... Production POST /weight-profiles → 500 SyntaxError. ## Root cause SQLAlchemy text() parser трактует ':weights' followed by '::jsonb' двусмысленно: - ':weights' — named param binding (должен стать %(weights)s) - '::jsonb' — PG cast operator Parser НЕ распознаёт :weights как параметр когда сразу следует :: cast. Result: psycopg видит literal ':weights' в SQL → syntax error. ``` LINE 5: ($1, $2, :weights::jsonb, $3, $4) ^ parameters: {user_id, profile_name, is_default, description} ↑ note: 'weights' missing — SQLAlchemy skipped it ``` ## Fix Replace ambiguous :weights::jsonb с CAST(:weights AS jsonb): - _INSERT (line 102) - _UPDATE dynamic 'weights = :weights::jsonb' (line 273) CAST() syntax не имеет ambiguity, SQLAlchemy корректно bind'ит :weights. ## Affected endpoints - POST /api/v1/admin/site-finder/weight-profiles (create) - PUT /api/v1/admin/site-finder/weight-profiles/{id} (update with weights) Refs: user report 2026-05-15 Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 08:12:18 +03:00
lekss361	e93bb6121e	feat(site-finder): D2 velocity-score from domrf_kn_sale_graph (#34 sub-PR 1/2) (#146) ... compute_velocity service queries competitor sales в радиусе 3км: - ST_MakePoint(longitude, latitude) — domrf_kn_objects не имеет geom column - JOIN domrf_kn_sale_graph за 6 мес (area_sq primary, realised*45 fallback) - Normalize vs ЕКБ-wide median → velocity_score 0..1 - confidence: high/medium/low (competitors_count + months_observed) - Top 5 sample competitors для UI Integration: analyze_parcel.response['velocity'] top-level field. Schema corrections vs spec: - obj_name → comm_name - region_code → region_cd - contracted (INT) → area_sq (м²) Tests: 102/102 pass. Vault: Module_Velocity_Service.md NEW. Closes #144 (sub-PR 2 frontend закроет #34) Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 01:24:53 +03:00
lekss361	76a88b2827	feat(site-finder): gate verdict aggregator — can-build-MKD signal (#32 G5 sub-PR 1/2) (#142) ... Pure-function aggregator collapses nspd_zoning / nspd_zouit_overlaps / nspd_engineering_nearby / nspd_dump в один GateVerdict TypedDict. Logic: - ПЗЗ не Ж-* → BLOCKER - ЗОУИТ sub=17 (инжен. охранная) → BLOCKER - Другие ЗОУИТ → WARNING - Нет инжен. сетей в 200м → WARNING - nspd_dump stale → source: nspd_dump_partial - Нет dump → can_build_mkd: 'unknown' Integration: новое поле gate_verdict в analyze_parcel response. Tests: 15/15 pass (mock-based). Vault: code/modules/Module_Gate_Verdict.md NEW. Closes #140 (sub-PR 2 frontend закроет #32) Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 00:59:53 +03:00
lekss361	5aea78c2d8	feat(site-finder): weight profiles service — Pydantic + CRUD (#114 sub-PR 2/4) (#137) ... Per #114 sub-PR 2/4. Pydantic v2 models + CRUD service. 14 passing tests. API: list/get/get_default/create/update/delete_profile + resolve_weights fallback to system defaults. Validation: ALLOWED_CATEGORIES guard + weight bounds [-2.0, 3.0]. Default-uniqueness в одной transaction (job_settings.py pattern). Vault: code/modules/Module_Weight_Profiles_Service.md. Refs: #114 Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 00:29:37 +03:00
lekss361	a3e7d21e3a	fix(pzz_loader): replace bare db.rollback() with SAVEPOINT per-row (#124) ... Per code review audit (May 14): transaction correctness bug в `sync_pzz_zones_to_db()` — inner `db.rollback()` в loop сбрасывал outer transaction на per-row failures → counts inserted/updated НЕ matched DB state. ## Fix Wrap UPSERT в loop через `db.begin_nested()` (SAVEPOINT): - Failure одной row → rollback только её savepoint, outer tx alive - Successfully inserted rows accumulate в outer tx → commit final - Outer try/except добавлен для unexpected error → rollback + log + raise - `finally: db.close()` preserved Pattern consistent с `domrf_kn.py` (где SAVEPOINT уже работает). ## Preserved - Function signature + return shape {fetched, inserted, updated, skipped} — unchanged - Logging contracts - skipped count теперь корректно включает DB-failure rows (consistent with prior behavior) ## Tests - No test_pzz_loader.py — skipped - ruff + AST passed ## Vault `fixes/Bug_Pzz_Loader_Missing_Savepoint_May14.md` — created (status: resolved). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-14 23:29:23 +03:00
lekss361	1c2f80a4b8	feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 FINAL) (#116) ... * feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 #4 FINAL) Замыкает Sprint 1.1 из #94 part 2 plan. После этого PR пользователь видит свежие НСПД данные в UI (frontend integration — отдельный PR). Backend (new app/services/site_finder/quarter_dump_lookup.py): - `derive_quarter_cad(cad_num)` — 3/4/5-сегмент → quarter (3-segment) - `get_quarter_dump_data(db, cad_num, parcel_wkt)` — main entrypoint: - Reads nspd_quarter_dumps row для derived quarter - Freshness threshold: 180 days - Missing/stale/harvest_error → trigger harvest_quarter.apply_async() fire- and-forget (lazy import против circular), return EMPTY_DUMP_RESULT - Fresh + parcel_wkt=None → metadata only (no spatial queries) - Fresh + geometry → 3 spatial queries via jsonb_array_elements + ST_Transform (3857→4326) + ST_Intersects / ST_DWithin - 3 private helpers: - `_get_zoning` — point-in-polygon parcel centroid vs territorial_zones, LIMIT 1 - `_get_zouit_overlaps` — все zouit_% layers пересекающиеся с parcel - `_get_engineering_nearby` — engineering_structures в 200m, sorted by distance - `EMPTY_DUMP_RESULT` module-level constant — DRY для no-dump fallback (used in get_quarter_dump_data internal + analyze_parcel try/except wrap) Backend (parcels.py): - Import EMPTY_DUMP_RESULT + get_quarter_dump_data - Call wrapped в try/except — если nspd_quarter_dumps недоступна (DB timeout / table missing) → EMPTY_DUMP_RESULT fallback вместо 500 (consistent с resilience pattern других optional fetches) - Response gets 4 new fields: - nspd_zoning: dict | None (G1 ПЗЗ — zone_code, zone_name, source) - nspd_zouit_overlaps: list[dict] (G3 — overlaps в parcel, per ЗОУИТ group) - nspd_engineering_nearby: list[dict] (I3 — engineering structures в 200m) - nspd_dump: dict (freshness metadata — available, fetched_at_utc, stale, harvest_triggered, total_features) Tests: 13 new в test_quarter_dump_lookup.py (mock-based, no real DB): - derive_quarter_cad 5 edge cases (3seg, 4seg, 5seg, invalid, whitespace) - get_quarter_dump no_row → harvest triggered - stale (>180d) → harvest triggered, stale=True - harvest_error row → retry harvest triggered - parcel_wkt=None → metadata only (1 DB call) - fresh + zoning extraction - fresh + zouit_overlaps list - fresh integration: все 4 keys present 47 pre-existing tests still pass. Code review (code-reviewer pre-push): MINOR, 0 blocking. Applied 2 of 4: - ✅ #1: try/except wrap around get_quarter_dump_data в analyze_parcel (защита от DB unavailability) + DRY через EMPTY_DUMP_RESULT module const - ✅ #2: removed redundant nspd_zoning.fetched_at_utc (DRY — freshness в nspd_dump.fetched_at_utc) - ⏭ Deferred (acceptable): #3 ad-hoc harvest_quarter retry cooldown для harvest_error rows (только при high traffic + persistent NSPD errors); #4 raw_props в response — tech debt, убрать вместе с frontend PR Performance note: 3 spatial queries per analyze adds ~10-50ms on typical ~100-feature quarter. Mitigation if quarters grow dense: materialized per-layer sub-table (отдельная DB issue). Closes Sprint 1.1 part of #94. Frontend rendering этих 4 полей — отдельный PR (next: #112 / #115 / #114). * fix(site-finder): address PR #116 auto-review M1-M5 M1 (mutation risk): replace EMPTY_DUMP_RESULT direct refs with _make_empty_result() factory. dict(...) shallow copy left nested nspd_dump shared by reference across concurrent requests — single mutation pollutes module sentinel for all subsequent calls. Now каждый caller gets independent dict. M2 (O(N) spatial scan): SELECT extended denormalized counts (territorial_zones_count, zouit_count, engineering_count). Each spatial helper accepts layer_counts and early-returns when count=0 — skips heavy jsonb_array_elements + ST_Transform/ST_Intersects scan entirely. Critical для quarters с 2000+ features. M4 (documentation): _trigger_harvest docstring describes known burst/no-dedup limitation + TODO Redis SETNX (отдельный PR). M5 (test fragility): _make_db_mock_with_spatial docstring describes positional-call contract — db.execute order (0=dump, 1=zoning, 2=zouit, 3=engineering) и зависимость от count-values. +4 new tests (17 total, all pass): - test_make_empty_result_returns_independent_copies (mutation safety) - test_make_empty_result_overrides - test_early_exit_all_counts_zero_no_spatial_queries - test_early_exit_partial_counts Per auto-review on 3068a9c. * fix(site-finder): rename _make_empty_result → make_empty_result (public) per PR #116 review M1 residual fix: parcels.py exception path использовал EMPTY_DUMP_RESULT singleton ref вместо factory. Сейчас readonly access, но нарушает documented invariant модуля. Rename `_make_empty_result` → `make_empty_result` (public API), import в parcels.py, использовать в try/except fallback. Каждый request получает независимый dict — никаких shared references. M4 (Redis SETNX dedup) + M5 (test fragility) — deferred per review, documented в code/issue. Acceptable trade-offs: - M4: UPSERT idempotency делает данные safe; burst-duplicate task'и тратят WAF traffic впустую но не повреждают данные. - M5: docstring contractually describes positional-call order. 17/17 tests pass. ruff/format clean. Per auto-review on aef8308. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-13 09:14:19 +03:00
lekss361	49fac806fc	feat(site-finder): auto-fetch cadastre geometry on-demand (#93) (#95) ... * feat(site-finder): auto-fetch cadastre geometry on-demand (#93) Когда пользователь вводит cad-номер которого нет в БД (cad_quarters_geom / cad_buildings / cad_parcels_geom), вместо 404 «Загрузи через NSPD geo» (dead- end для non-admin) — теперь backend автоматически инициирует NSPD fetch, frontend показывает loading state с polling /fetch-status каждые 2с. Backend: - Новый модуль app/services/site_finder/cadastre_fetch.py с helpers: find_or_enqueue_fetch (atomic check + enqueue с дедупликацией по cad), fetch_status (smart polling endpoint — отличает not_in_nspd от failed), detect_thematic_id (3-сегм quarter / 4-сегм parcel / 5-сегм building), validate_cad_format. - Reuse: enqueue_geo_job + process_nspd_geo_job (workers/tasks/nspd_geo). source_kind='auto_on_demand' отличает от bulk; rate_ms=200, priority=9. - POST /parcels/{cad}/analyze graceful fallback: inline await до 25s (fast path), затем 202 + job_id + eta_seconds для polling, либо 400/404/503 в зависимости от статуса (с Retry-After 60s на 503). - GET /parcels/{cad}/fetch-status новый endpoint для polling. Frontend: - useSiteAnalysis расширен: fetchingState + cancel(). POST analyze + при 202 polling каждые 2с (max 60 итераций = 2 мин cap). status=ready → re-trigger analyze; not_in_nspd/failed/invalid_format → typed errors. - apiFetchWithStatus<T> + HTTPError в lib/api.ts — status-aware variant для 200 vs 202. - FetchingState.tsx: animated spinner, progress bar (linear до etaSeconds, asymptote после), elapsed counter, cancel button. Светло-голубая scheme отличается от обычного pending skeleton. - site-finder/page.tsx: FetchingState когда fetchingState активен; обычный pending skeleton — только при первичном analyze без 202. Edge cases (per #93 acceptance): - cad валидный в НСПД, fetch <25s → inline 200 OK - cad валидный, fetch >25s → 202 + frontend polls → ready → analyze - cad валидный, не в НСПД → 404 с понятным сообщением + формат hint - cad invalid format → 400 + формат hint - NSPD rate-limited / failed → 503 + Retry-After 60s - Параллельные запросы на тот же cad → один job, оба клиента poll'ят (дедуп через find_active_on_demand_job). Closes #93. * fix(site-finder): address PR #95 auto-review minor feedback Backend (cadastre_fetch.py): 1. (race condition) Advisory lock `pg_try_advisory_xact_lock(hashtext(cad_num))` обёрнут вокруг шагов "check active job → enqueue" в find_or_enqueue_fetch. Lock transaction-scoped, released at COMMIT. Параллельные запросы на тот же cad: первый получает lock и enqueue; второй lock=false → re-check active job (уже виден после первого COMMIT) → возвращает тот же job_id. Docstring обновлён, упоминание SELECT FOR UPDATE удалено. Backend (parcels.py): 3. (threadpool exhaustion) _INLINE_FETCH_WAIT_S снижен 25 → 15s с подробным комментарием: tradeoff про default Starlette anyio threadpool (40 slots) и concurrent burst saturation. 15s баланс: НСПД avg 5-15s для quarter, ~70% fast path; остальные 30% получают 202 + polling без блока. Data (87_on_demand_indexes.sql): 2. (missing index) New migration: - `nspd_geo_targets_cad_num_idx ON nspd_geo_targets(cad_num)` — для find_active/recent_completed_job (existing UNIQUE composite не покрывает WHERE cad_num=:c). - `nspd_geo_jobs_source_status_idx ON nspd_geo_jobs(source_kind, status)` composite для filter auto_on_demand + queued/running. Idempotent (CREATE INDEX IF NOT EXISTS), не блокер при текущем размере, критично при росте on-demand traffic. Frontend (useSiteAnalysis.ts): 4. (UI flicker) setFetchingState(null) перемещён ПОСЛЕ `await second apiFetch`. Иначе между clear и resolve есть момент когда mutation isPending=true + fetchingState=null → пустой экран ~1 RTT. NOT addressed (rebuttal): 5. (Tailwind convention) — проверил: в проекте нет ни globals.css, ни @tailwind directives. ВСЕ существующие site-finder components используют inline styles (ConfidenceBadge, GeotechRiskBlock, ScoreBreakdownPanel etc). Tailwind в deps но не wired up. Keep inline styles для consistency. 6. (animate-spin) — требует Tailwind globals (см. #5). `<style jsx>` keyframes — built-in Next.js, работает. Per auto-review on 2252236. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-12 09:02:17 +03:00
lekss361	49eadeb9ce	fix(pzz-sync): follow_redirects=True (PKK6 returns 301)	2026-05-11 22:50:46 +03:00
lekss361	1c1ecad8b8	fix(pzz-sync): disable SSL verify for Rosreestr PKK6 (self-signed cert chain)	2026-05-11 22:44:53 +03:00
lekss361	8be539ddc6	feat(site-finder): ПЗЗ territorial zones from Rosreestr PKK6 + zoning in analyze	2026-05-11 21:51:51 +03:00
lekss361	5b03d6d799	feat(site-finder): isochrones UI + networks VKH + OSM substations	2026-05-11 21:48:21 +03:00
lekss361	6ce634a9b9	feat(site-finder): utilities (power/pipeline) + fix Внешние факторы layout	2026-05-11 21:38:20 +03:00
lekss361	2855a01ca7	feat(site-finder): v3.5 - seasonal weather + hydrology + geotech risk	2026-05-11 21:32:07 +03:00
lekss361	7900dc5238	fix(site-finder): WKT LINESTRING needs commas + rosreestr_deals real column names	2026-05-11 20:58:49 +03:00
lekss361	1e9d32ee3c	feat(site-finder): v3.2 — noise + air quality + wind analytics	2026-05-11 20:42:37 +03:00
lekss361	f419900968	feat(site-finder): v3.1 — cad_parcels_geom, analyze fallback, POI lat/lon, OSM expand ... - nspd_geo: add _save_parcel() for thematic_id=1 → cad_parcels_geom (UPSERT, ST_Transform from Web Mercator); _persist_target now handles 1/2/5 - parcels.py: analyze endpoint geom lookup extended with cad_parcels_geom as 3rd fallback source (after cad_quarters_geom, cad_buildings); both SELECT and WKT subqueries updated - parcels.py: POI score_breakdown items now include lat/lon for map markers - poi_loader: OSM_CATEGORIES expanded — college+university→school, hypermarket→shop_supermarket; coverage +3 tag pairs	2026-05-11 19:52:19 +03:00
lekss361	3aeda297ee	fix(poi-sync): split Overpass query per category (504 Gateway Timeout)	2026-05-11 18:54:28 +03:00
lekss361	93f13795bb	fix(poi-sync): add User-Agent header — Overpass returns 406 for python-httpx default UA	2026-05-11 18:46:47 +03:00
lekss361	6dea34186b	feat(site-finder): OSM POI loader + Celery weekly sync ... - New poi_loader.py: Overpass QL query for 13 POI categories in EKB bbox, UPSERT into osm_poi_ekb with soft 2-year filter tracking (skipped_old counter) - New poi_sync.py Celery task wrapping sync_poi_to_db() - celery_app.py: include poi_sync module, add poi-sync-weekly beat entry (Mon 03:00 MSK)	2026-05-11 18:08:34 +03:00
lekss361	b95559eac9	init	2026-04-25 13:45:19 +03:00