91 commits

Author	SHA1	Message	Date
lekss361	5a7d405c77	Merge pull request 'feat(nspd): TIER 4 opportunity + red lines (#94 PR2 of 4)' (#220) from feat/94-pr2-opportunity-redlines into main	2026-05-16 17:09:52 +00:00
lekss361	fdb54834e5	fix(nspd): rename migration 89→98, fix red lines ST_Area→ST_Length (#220) ... - data/sql/98_*: rename from 89 to avoid collision with existing 89_drop_dead_brin - _get_red_lines: ST_Length(ST_Intersection(planar)::geography) instead of ST_Area(ST_Intersection(::geography, ::geography)) — fixes PostGIS 3.4 tolerance error and returns correct non-zero length for LINESTRING intersections - Rename intersection_area_sqm → intersection_length_m across schema, TS types, frontend component, and tests; add test_get_red_lines_db_exception_returns_empty Addresses review-bot feedback on PR #220.	2026-05-16 19:33:14 +03:00
lekss361	ed3c128528	feat(nspd): TIER 4 opportunity layers + red lines (#94 PR2 of 4) ... - NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt) + QuarterDump.opportunity field - nspd_sync: harvest_quarter accepts include_opportunity, denorm cols has_auction_parcels + opportunity_count in UPSERT - quarter_dump_lookup: - _get_opportunity_parcels (sort by distance, early-exit on count=0) - _get_red_lines (query existing dump.red_lines core path, layer='red_lines') - SQL 89_*: has_auction_parcels + opportunity_count + partial index - Pydantic: OpportunityParcel + RedLine schemas - /analyze: nspd_opportunity_parcels + nspd_red_lines fields - Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration - Tests: 28 total (11 PR1 + 17 PR2), all pass Red lines uses existing core harvest path (layer 879243 already in dump.red_lines from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed. Part of #94	2026-05-16 19:06:22 +03:00
lekss361	fbb76812f4	fix(recommend): cad_buildings.floors is INTEGER, not TEXT ... PROD 500 on /api/v1/analytics/recommend/mix: psycopg.errors.UndefinedFunction: operator does not exist: integer ~ unknown LINE 19: AND ((cb.floors ~ '^[0-9]+$' AND cb.f... _district_cadastre_baseline use regex \ on cb.floors. Schema is INTEGER (verified information_schema), so the regex defence against text values ('1-2', '2-3' etc) is misplaced — simplify to floors >= 3 with NULL safety via OR purpose ILIKE '%многокв%' fallback.	2026-05-16 18:29:35 +03:00
lekss361	1595157ebf	feat(nspd): enable TIER 3 risk zones harvest + extract in analyze (#94 PR1) ... - harvest_stale_quarters fanout passes include_risks=True to all queued tasks - _get_risk_zones() with ST_Intersects + ST_Area intersection calc - _extract_features_by_layer() generic helper for future PR2+ reuse - RiskZone Pydantic schema; /analyze returns nspd_risk_zones list - NspdRiskZonesBlock component with severity color-coding (red/yellow/orange) - 11 unit tests for extract/intersect/null-area/db-exception paths Part of #94 (TIER 3 risk layers — flooding, landslide, burns, erosion etc)	2026-05-16 16:11:59 +03:00
lekss361	dbf31cc911	feat(etl): objective_complex_mapping backfill via pg_trgm (#203) (#214)	2026-05-16 12:47:09 +00:00
lekss361	adc45f5724	feat(permits): ЕКБ РНС/РВЭ xlsx parser + schema + ETL (#105 PR A) (#211)	2026-05-16 11:41:21 +00:00
lekss361	efa66a9f68	feat(analytics): mv_quarter_price_per_m2 MV + refresh helper (#33 D1 PR A) (#209)	2026-05-16 11:12:01 +00:00
lekss361	092976f656	feat(site-finder): inline POI weights pass-through в /analyze (#201 Phase 1) (#206)	2026-05-16 11:00:41 +00:00
lekss361	256909d28b	feat(layouts): PDF ТЗ endpoint + BestLayoutsBlock UI (#113 PR D) (#199)	2026-05-16 09:53:05 +00:00
lekss361	ad16fc0e42	feat(parcels): best-layouts endpoint + service (#113 PR C) (#196)	2026-05-16 09:10:06 +00:00
lekss361	6aa22ef8ba	feat(analytics): mv_layout_velocity MV + refresh helper (#113 PR B) (#194)	2026-05-16 08:26:59 +00:00
lekss361	e673811a20	feat(parcels): layout signature schemas + extractor (#113 PR A) (#193)	2026-05-16 07:52:34 +00:00
lekss361	4deb1b0284	feat(parcels): connection-points endpoint (Forgejo #115 Phase 1 backend) (#190)	2026-05-16 05:58:27 +00:00
lekss361	e561df1b55	feat(parcels): competitors endpoint (Forgejo #112 Phase 1 backend) (#191)	2026-05-15 22:21:05 +00:00
lekss361	4d492ebc50	perf(cadastre): freshness skip + grid-walk heartbeat + log noise reduction (#187) ... * perf(cadastre): freshness skip + grid-walk heartbeat + reduce WMS log noise Three operational improvements after ekb_full v3 completed at 99.1%: Fix H — per-quarter freshness skip (scrape_cadastre.py): - enqueue_cadastre_harvest queries cad_quarter_stats.fetched_at per quarter - Skip quarters where fetched_at >= NOW() - skip_fresh_hours (default 1h) - extra_config.skip_fresh_hours=0 disables skip (full re-fetch) - Saves ~30k NSPD calls on repeated runs of same scope Fix I — heartbeat in grid-walk loop (bulk_harvest.py:_grid_walk_category): - 225 cells x 70-300ms = 15-75s heartbeat silence - Pass update_progress callback to _grid_walk_category - Call every heartbeat_every (50) cells with phase progress info - Prevents cleanup_zombies false-positive cancellation Fix J — NSPD WMS HTTP 500 noise to debug (bulk_harvest.py:_grid_walk_category): - NSPD WMS endpoint frequently returns HTTP 500 (server-side issue) - Was: logger.warning each, 200+ warnings per quarter - Now: logger.debug (visible via --log-level=DEBUG) 40 tests passing. * fix(cadastre): parse NSPD DD.MM.YYYY dates (Fix K — job 18 root cause) Job 18 (manual_list recovery 11 quarters) crashed every quarter with: psycopg.errors.DatetimeFieldOverflow: date/time field value out of range: '13.03.2023' NSPD returns registration_date in mixed formats: - DD.MM.YYYY for Сооружения / ОНС / ЕНК / ЗОУИТ - YYYY-MM-DD ISO for ЗУ / Здания - YYYY-MM-DD HH:MM:SS with time PG CAST(:date AS date) accepts ISO only — DD.MM.YYYY crashes. Add _parse_nspd_date helper: handles both formats + None/empty. Apply in upsert_construction, upsert_onc, upsert_enk, upsert_zouit, upsert_parcel for registration_date / build_record_registration_date / legal_act_date. 3 new unit tests for the helper (43 tests total). * fix(cadastre): address bot blockers — pivot Fix H to raw_targets + tests Bot review on PR #187 caught: 1. cadastre_jobs has no extra_config column → Fix H crashed on KeyError 2. Missing tests for Fix H + Fix I 3. _parse_nspd_date didn't validate date ranges Changes: - Pivot Fix H: read skip_fresh_hours from raw_targets JSONB (existing column), no schema migration needed - _parse_nspd_date: validate via date.fromisoformat() — rejects 2023-13-45 - Add 3 tests for Fix H in backend/tests/workers/test_scrape_cadastre.py - Add 2 tests for Fix I heartbeat callbacks in test_cadastre_bulk.py - Add 1 test for parse_nspd_date range validation (was: only format) 49 tests passing. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 23:02:37 +03:00
lekss361	d104222643	fix(cadastre,nspd_sync): SQL syntax bugs surfaced by ekb_full v2 (#186) ... Three distinct SQL syntax errors caught from worker logs: 1. **upsert_zouit** (bulk_harvest.py:940) — trailing comma after 4326 in ST_Transform(..., 3857), 4326,) → 'syntax error at or near ")"'. Hit on 8+ quarters in 0108 block (66:41:0108055/61/69/77 etc). Likely introduced by ruff-format auto-comma on closing paren. 2. **nspd_sync** (nspd_sync.py:120) — `:fetched_at_utc::timestamptz` psycopg double-colon trap → 'syntax error at or near ":"'. Crashed legacy on-demand harvest_quarter called from analyze. 3. **job_settings** (job_settings.py:224) — `:extra_config::jsonb` same double-colon trap in admin settings UPDATE. All converted to CAST(:x AS type) pattern (psycopg v3 safe). 40 tests still passing. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 19:18:43 +03:00
lekss361	5f8df655cc	fix(cadastre): cad_parcels.geom Polygon -> MultiPolygon (migration 93) (#185) ... NSPD returns MultiPolygon for Многоконтурный участок (e.g. 66:41:0105017:4) which crashed upsert_parcel — schema was strict POLYGON. Migration 93: ALTER COLUMN geom Polygon -> MultiPolygon USING ST_Multi(geom), DROP+RECREATE GIST index + back-compat VIEW cad_parcels_geom. upsert_parcel SQL wraps ST_Transform in ST_Multi() — coerces Polygon to MultiPolygon. NSPD returns both types (simple + multi-contour parcels). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 18:52:38 +03:00
lekss361	e3359c4f03	fix(cadastre): filter Point/LineString geometry in upsert_parcel/zouit (#184) ... cad_parcels.geom is geometry(Polygon, 4326) — strict schema. NSPD occasionally returns Point geometry for parcels without detailed boundary, causing INSERT failure: psycopg.errors.InvalidParameterValue: Geometry type (Point) does not match column type (Polygon) This killed pilot v8 at quarter 66:41:0104002 — worker autoretry exhausted, job hung at 25/50 (heartbeat stale). Fix: filter geometry.type at Python level in upsert_parcel + upsert_zouit (same pattern as upsert_quarter_geom_from_feature). Non-Polygon geometry → geom=NULL, raw_props preserved. Tables with permissive GEOMETRY schema (cad_buildings, cad_constructions, cad_enk, cad_oncs) unaffected — they accept any geometry type. Tests: 3 new (Point parcel → geom=NULL, Polygon happy path, LineString zouit). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 18:37:29 +03:00
lekss361	e695bca1e4	fix(cadastre): remove shared phase_state early-exit (race condition) (#183) ... cadastre_jobs.phase_state is shared between ALL parallel Celery workers of the same job. The early-exit triggered as soon as the FIRST worker wrote phase=done via progress_cb — all subsequent workers (reading same shared row) bailed without work. Effect: pilot v7 made 5 NSPD requests for 50 quarters (=0.1/quarter). Full ekb_full job #8: 4950 req / 2408 quarters = 2/quarter — 95% workers early-exited without doing snapshot phase. Root cause of 1.6% parcels coverage. Fix A/B/C only partially helped because most workers never reached Phase 1/1.5. Solution: delete the early-exit. Idempotency is guaranteed by ON CONFLICT DO UPDATE in every upsert_* — re-enqueued tasks write same rows without duplicates. Closes part of #168 (root-cause fix for race condition revealed by #182 metrics). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 17:41:27 +03:00
lekss361	057f8891dc	fix(cadastre): count snapshot+per-cat requests in cadastre_jobs.requests_count (#182) ... scrape_cadastre.py previously incremented requests_count only by grid_walk_requests, so snapshot-only quarters (and the entire ekb_full job after PR #179 skips grid-walk on broken geom) reported 0 NSPD requests — misleading for observability. Add HarvestResult.snapshot_requests counter (Phase 1 search + Phase 1.5 per-cat probes) and a total_requests property. Update Celery task to write total_requests into cadastre_jobs.requests_count. Tests added for the new property + Phase 1 snapshot_requests=1 assertion. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 17:17:01 +03:00
lekss361	905ef443fc	fix(cadastre): auto-heal cad_quarters_geom from CAT_QUARTER_STATS feature (#181) ... 72% of EKB quarters (1735/2408) have broken micro-precision geom in cad_quarters_geom. NSPD returns valid quarter polygon in CAT_QUARTER_STATS (36381) feature but Phase 4 was only consuming stats. Add upsert_quarter_geom_from_feature: CTE-based UPDATE that transforms 3857 polygon to 4326 and writes only when existing geom is NULL or broken (bbox width outside 100-10000m), verified via SQL-side sanity check. Wire into Phase 4 with begin_nested savepoint: malformed NSPD GeoJSON (self-intersecting ring) rollbacks only geom UPDATE, preserves upsert_quarter_stats. Closes part of #168 (follow-up data quality fix for #179). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 17:16:39 +03:00
lekss361	d5b57ccbf1	feat(cadastre): per-category NSPD snapshot — Phase 1.5 (ZOUIT/ENK) (#180) ... Switch base search_by_quarter from thematicSearchId=1 to thematic=1 (includes ZOUIT in features). Add optional category_id param that routes ZOUIT (36940/469039/469040/469042) -> thematicSearchId=5, ENK (39663) -> thematicSearchId=15. Add Phase 1.5 in harvest_quarter: for each PER_CATEGORY_PROBE_CATS category present in meta_counts (meta_total>0), run dedicated per-category search call. Skips if category in GRID_WALK_CATS and bbox is valid (grid-walk gives better coverage). Curl-probe confirmed NSPD API caps at 20/category with no pagination; categoryId param is server-ignored; thematic=1 strictly better than thematicSearchId=1 (includes ZOUIT). Closes part of #168 (ZOUIT/ENK coverage gap). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 16:53:49 +03:00
lekss361	3beb75d68a	fix(cadastre): skip grid-walk for broken quarter geometries (#168) (#179) ... * fix(cadastre): skip grid-walk for broken quarter geometries (#168) Pilot v5 (50 quarters) разоблачил что cad_quarters_geom имеет 1735/2408 (72%) ЕКБ rows с broken micro-precision geom (width ~0.01м instead of 200-4000м). Causes: - grid-walk WMS receives bbox ~1mm × 1mm → 500 ServiceException на all 450 req - One quarter hung 2+ min (silent failure in async loop) Real fix: re-import cad_quarters_geom geometry from NSPD (category 36381). Tracked в follow-up issue. Quick fix (this PR): sanity-check quarter_bbox_3857 — return None если width ИЛИ height not in [100m, 10000m]. Bulk_harvest_quarter then skips grid-walk для broken quarters, только snapshot phase запускается (20 per cat). After this fix: - All 50 pilot quarters получают snapshot (parcels/buildings/etc up to 20 each) - Grid-walk runs only для 673/2408 healthy quarters → no 500 cascades, no hangs * test(cadastre): add quarter_bbox_3857 boundary tests (PR #179 bot blocker) - healthy bbox 750m -> returns tuple - width <100m (broken micro-precision) -> None - width >10000m (oversized) -> None - height <100m (symmetry) -> None - row missing (quarter not in cad_quarters_geom) -> None --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 16:25:06 +03:00
lekss361	6dfff5e046	fix(cadastre): CAST :geom AS text — psycopg AmbiguousParameter (#168) (#178) ... Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 16:02:03 +03:00
lekss361	5da66bbc93	feat(cadastre): bulk_harvest worker + grid-walker + admin API (#168 PR3/5) (#171) ... * feat(cadastre): bulk_harvest_quarter Celery task + grid-walker + saga state (#168 PR3/5) Add bulk cadastre harvest pipeline: - services/cadastre/bulk_harvest.py: async harvest_quarter() orchestrator (4 phases) + 7 upsert helpers (parcels/buildings/constructions/oncs/enks/zouit/quarter_stats) using CAST(:x AS jsonb) pattern, begin_nested() SAVEPOINT per grid-walk upsert - services/cadastre/grid_geometry.py: quarter_bbox_3857 (PostGIS ST_Extent) + generate_grid_click_points (15x15 = 225 sub-bbox grid) - workers/tasks/scrape_cadastre.py: bulk_harvest_quarter_task (acks_late=True, dont_autoretry_for NspdBulkWafError), enqueue_cadastre_harvest, cleanup_zombies - api/v1/admin_cadastre.py: 5 endpoints behind AdminTokenAuth — create/list/get/cancel/resume - Tests: 13 service unit + 8 API tests * fixup(cadastre): drop importorskip (PR2 merged) — imports now top-level (#168 PR3) * fixup(cadastre): tile_size unified param + register slow marker (#168 PR3) Blocker #1: rename tile_width/tile_height → tile_size in generate_grid_click_points. Callers in bulk_harvest.py and test_cadastre_bulk.py used tile_size; def had tile_width+tile_height → TypeError at runtime. Blocker #2 (10 failures in test_admin_cadastre.py) was side-effect of #1: TypeError at import chain (app.main → admin_cadastre → bulk_harvest) broke FastAPI app load → dependency_overrides AttributeError. Now resolved. Also register `slow` pytest marker in pyproject.toml to suppress PytestUnknownMarkWarning. * fixup(cadastre): use importlib.util.find_spec instead of import (#168 PR3) Root cause Blocker #2: `import app.workers.tasks.scrape_cadastre` at module top-level REBOUND `app` from FastAPI instance (line 14) to the Python package. All subsequent `app.dependency_overrides[get_db] = ...` failed with AttributeError because `app` was now the namespace module. Fix: probe module availability with importlib.util.find_spec — does not import or rebind names. FastAPI `app` instance stays intact. All 28 cadastre tests pass locally. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 13:31:32 +03:00
lekss361	d7fbaa0528	feat(velocity): per-room-bucket breakdown in VelocityResult (#163) ... * fix(velocity): :window_interval::interval cast syntax — same bug as PR #152 SQLAlchemy text() parser confuses ':window_interval' (named param) followed by '::interval' (cast operator) — exactly как было в PR #152 с :weights::jsonb. Result: psycopg видит literal ':window_interval' в SQL → syntax error → exception caught by velocity catch → return None → UI shows null. ## Fix ':window_interval::interval' → 'CAST(:window_interval AS interval)' (2 мест: sales_rows query + _get_ekb_median percentile). Pre-push code-reviewer должен был catch — добавим в feedback что для каждого text() SQL grep ':[a-z]*::' before push. Refs: PR #158 deploy verify, e2e velocity = null root cause * feat(velocity): add per-room-bucket breakdown to VelocityResult Add third SQL query (bucket_rows) aggregating deals_total_count by room_bucket ('студия','1','2','3','4+') across mapped competitors. New fields: - VelocityResult.by_room_bucket: aggregate {units, sqm, complexes_count} - sample_competitors[].by_room_bucket: {bucket: units} Bucket query wrapped в SAVEPOINT — failure degrades gracefully к empty dict без abort outer tx. CAST(:window_interval AS interval) pattern per PR #160 lesson. Test coverage: 3 new tests (aggregation, empty fallback, sample entries). Closes part of #161 --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 10:09:03 +03:00
lekss361	46584d7c75	fix(velocity): :window_interval::interval cast syntax — same bug as PR #152 (#160) ... SQLAlchemy text() parser confuses ':window_interval' (named param) followed by '::interval' (cast operator) — exactly как было в PR #152 с :weights::jsonb. Result: psycopg видит literal ':window_interval' в SQL → syntax error → exception caught by velocity catch → return None → UI shows null. ## Fix ':window_interval::interval' → 'CAST(:window_interval AS interval)' (2 мест: sales_rows query + _get_ekb_median percentile). Pre-push code-reviewer должен был catch — добавим в feedback что для каждого text() SQL grep ':[a-z]*::' before push. Refs: PR #158 deploy verify, e2e velocity = null root cause Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 09:46:01 +03:00
lekss361	8fa1d005fd	feat(velocity): migrate D2 sales source → objective_corpus_room_month (#158) ... * feat(velocity): migrate D2 sales source from stale sale_graph to objective_corpus_room_month Closes #156 * fix(velocity): cross-stack contract + NULL safety per pre-push review 3 pre-push code-reviewer findings fixed: 1. TS union extended ('objective'|'sale_graph'|'rosreestr_fallback') + UI conditions handle both objective и sale_graph как valid sources. 2. COALESCE(deals_total_vol_m2, deals_total_count * 45.0) — NULL safety for DKP-only rows (vol_m2 nullable, count > 0). 3. room_bucket parking filter — verified false positive (все 5 buckets apartments: студия/1/2/3/4+). Refs: PR #157 pre-push code-reviewer * fix(types): add 'objective' to velocity_source union (cherry-pick miss) Cherry-pick от закрытого PR #157 потерял этот файл. Frontend type-check fails с 'no overlap' error. Restoring. Refs: PR #158 frontend CI failure --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 09:34:00 +03:00
lekss361	c39b53cf53	fix(site-finder): cascade InFailedSqlTransaction — velocity internal rollback (#155) ... * fix(site-finder): db.rollback() в velocity catches + SAVEPOINT zoning/success_rec velocity.py 3 internal try/except возвращали None БЕЗ db.rollback(). PR #154 SAVEPOINT не помог — exception не propagates из velocity. Postgres tx aborted внутри, cascade InFailedSqlTransaction на следующих queries. Fix: - velocity.py: db.rollback() в 3 catch блоках - parcels.py: SAVEPOINT обёртки для zoning + success_recommendation Refs: user reports 2026-05-15 cascade 500 * fix(velocity): SAVEPOINT per query (not db.rollback) — correct SQLAlchemy 2.0 pattern Per PR #155 bot review — мой db.rollback() в catches был НЕПРАВИЛЬНЫЙ. Session.rollback() = ENTIRE outer tx (не savepoint). Outer SAVEPOINT context становится orphaned → InvalidRequestError на __exit__. Correct fix: wrap каждый db.execute в свой with db.begin_nested(): - Failure → __exit__ propagates exception → SAVEPOINT rolls back - Outer tx остаётся clean - velocity returns None gracefully → caller продолжает Все 3 SQL queries в velocity.py теперь SAVEPOINT-wrapped: - compute_velocity: competitor_query - compute_velocity: sales_rows query - _get_ekb_median: median query db.rollback() removed из всех 3 catches. Refs: PR #155 bot review --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 08:52:38 +03:00
lekss361	af39e95f83	fix(site-finder): SQLAlchemy syntax error :weights::jsonb → CAST(:weights AS jsonb) (#152) ... Production POST /weight-profiles → 500 SyntaxError. ## Root cause SQLAlchemy text() parser трактует ':weights' followed by '::jsonb' двусмысленно: - ':weights' — named param binding (должен стать %(weights)s) - '::jsonb' — PG cast operator Parser НЕ распознаёт :weights как параметр когда сразу следует :: cast. Result: psycopg видит literal ':weights' в SQL → syntax error. ``` LINE 5: ($1, $2, :weights::jsonb, $3, $4) ^ parameters: {user_id, profile_name, is_default, description} ↑ note: 'weights' missing — SQLAlchemy skipped it ``` ## Fix Replace ambiguous :weights::jsonb с CAST(:weights AS jsonb): - _INSERT (line 102) - _UPDATE dynamic 'weights = :weights::jsonb' (line 273) CAST() syntax не имеет ambiguity, SQLAlchemy корректно bind'ит :weights. ## Affected endpoints - POST /api/v1/admin/site-finder/weight-profiles (create) - PUT /api/v1/admin/site-finder/weight-profiles/{id} (update with weights) Refs: user report 2026-05-15 Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 08:12:18 +03:00
lekss361	e93bb6121e	feat(site-finder): D2 velocity-score from domrf_kn_sale_graph (#34 sub-PR 1/2) (#146) ... compute_velocity service queries competitor sales в радиусе 3км: - ST_MakePoint(longitude, latitude) — domrf_kn_objects не имеет geom column - JOIN domrf_kn_sale_graph за 6 мес (area_sq primary, realised*45 fallback) - Normalize vs ЕКБ-wide median → velocity_score 0..1 - confidence: high/medium/low (competitors_count + months_observed) - Top 5 sample competitors для UI Integration: analyze_parcel.response['velocity'] top-level field. Schema corrections vs spec: - obj_name → comm_name - region_code → region_cd - contracted (INT) → area_sq (м²) Tests: 102/102 pass. Vault: Module_Velocity_Service.md NEW. Closes #144 (sub-PR 2 frontend закроет #34) Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 01:24:53 +03:00
lekss361	76a88b2827	feat(site-finder): gate verdict aggregator — can-build-MKD signal (#32 G5 sub-PR 1/2) (#142) ... Pure-function aggregator collapses nspd_zoning / nspd_zouit_overlaps / nspd_engineering_nearby / nspd_dump в один GateVerdict TypedDict. Logic: - ПЗЗ не Ж-* → BLOCKER - ЗОУИТ sub=17 (инжен. охранная) → BLOCKER - Другие ЗОУИТ → WARNING - Нет инжен. сетей в 200м → WARNING - nspd_dump stale → source: nspd_dump_partial - Нет dump → can_build_mkd: 'unknown' Integration: новое поле gate_verdict в analyze_parcel response. Tests: 15/15 pass (mock-based). Vault: code/modules/Module_Gate_Verdict.md NEW. Closes #140 (sub-PR 2 frontend закроет #32) Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 00:59:53 +03:00
lekss361	5aea78c2d8	feat(site-finder): weight profiles service — Pydantic + CRUD (#114 sub-PR 2/4) (#137) ... Per #114 sub-PR 2/4. Pydantic v2 models + CRUD service. 14 passing tests. API: list/get/get_default/create/update/delete_profile + resolve_weights fallback to system defaults. Validation: ALLOWED_CATEGORIES guard + weight bounds [-2.0, 3.0]. Default-uniqueness в одной transaction (job_settings.py pattern). Vault: code/modules/Module_Weight_Profiles_Service.md. Refs: #114 Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 00:29:37 +03:00
lekss361	a3e7d21e3a	fix(pzz_loader): replace bare db.rollback() with SAVEPOINT per-row (#124) ... Per code review audit (May 14): transaction correctness bug в `sync_pzz_zones_to_db()` — inner `db.rollback()` в loop сбрасывал outer transaction на per-row failures → counts inserted/updated НЕ matched DB state. ## Fix Wrap UPSERT в loop через `db.begin_nested()` (SAVEPOINT): - Failure одной row → rollback только её savepoint, outer tx alive - Successfully inserted rows accumulate в outer tx → commit final - Outer try/except добавлен для unexpected error → rollback + log + raise - `finally: db.close()` preserved Pattern consistent с `domrf_kn.py` (где SAVEPOINT уже работает). ## Preserved - Function signature + return shape {fetched, inserted, updated, skipped} — unchanged - Logging contracts - skipped count теперь корректно включает DB-failure rows (consistent with prior behavior) ## Tests - No test_pzz_loader.py — skipped - ruff + AST passed ## Vault `fixes/Bug_Pzz_Loader_Missing_Savepoint_May14.md` — created (status: resolved). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-14 23:29:23 +03:00
lekss361	1c2f80a4b8	feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 FINAL) (#116) ... * feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 #4 FINAL) Замыкает Sprint 1.1 из #94 part 2 plan. После этого PR пользователь видит свежие НСПД данные в UI (frontend integration — отдельный PR). Backend (new app/services/site_finder/quarter_dump_lookup.py): - `derive_quarter_cad(cad_num)` — 3/4/5-сегмент → quarter (3-segment) - `get_quarter_dump_data(db, cad_num, parcel_wkt)` — main entrypoint: - Reads nspd_quarter_dumps row для derived quarter - Freshness threshold: 180 days - Missing/stale/harvest_error → trigger harvest_quarter.apply_async() fire- and-forget (lazy import против circular), return EMPTY_DUMP_RESULT - Fresh + parcel_wkt=None → metadata only (no spatial queries) - Fresh + geometry → 3 spatial queries via jsonb_array_elements + ST_Transform (3857→4326) + ST_Intersects / ST_DWithin - 3 private helpers: - `_get_zoning` — point-in-polygon parcel centroid vs territorial_zones, LIMIT 1 - `_get_zouit_overlaps` — все zouit_% layers пересекающиеся с parcel - `_get_engineering_nearby` — engineering_structures в 200m, sorted by distance - `EMPTY_DUMP_RESULT` module-level constant — DRY для no-dump fallback (used in get_quarter_dump_data internal + analyze_parcel try/except wrap) Backend (parcels.py): - Import EMPTY_DUMP_RESULT + get_quarter_dump_data - Call wrapped в try/except — если nspd_quarter_dumps недоступна (DB timeout / table missing) → EMPTY_DUMP_RESULT fallback вместо 500 (consistent с resilience pattern других optional fetches) - Response gets 4 new fields: - nspd_zoning: dict | None (G1 ПЗЗ — zone_code, zone_name, source) - nspd_zouit_overlaps: list[dict] (G3 — overlaps в parcel, per ЗОУИТ group) - nspd_engineering_nearby: list[dict] (I3 — engineering structures в 200m) - nspd_dump: dict (freshness metadata — available, fetched_at_utc, stale, harvest_triggered, total_features) Tests: 13 new в test_quarter_dump_lookup.py (mock-based, no real DB): - derive_quarter_cad 5 edge cases (3seg, 4seg, 5seg, invalid, whitespace) - get_quarter_dump no_row → harvest triggered - stale (>180d) → harvest triggered, stale=True - harvest_error row → retry harvest triggered - parcel_wkt=None → metadata only (1 DB call) - fresh + zoning extraction - fresh + zouit_overlaps list - fresh integration: все 4 keys present 47 pre-existing tests still pass. Code review (code-reviewer pre-push): MINOR, 0 blocking. Applied 2 of 4: - ✅ #1: try/except wrap around get_quarter_dump_data в analyze_parcel (защита от DB unavailability) + DRY через EMPTY_DUMP_RESULT module const - ✅ #2: removed redundant nspd_zoning.fetched_at_utc (DRY — freshness в nspd_dump.fetched_at_utc) - ⏭ Deferred (acceptable): #3 ad-hoc harvest_quarter retry cooldown для harvest_error rows (только при high traffic + persistent NSPD errors); #4 raw_props в response — tech debt, убрать вместе с frontend PR Performance note: 3 spatial queries per analyze adds ~10-50ms on typical ~100-feature quarter. Mitigation if quarters grow dense: materialized per-layer sub-table (отдельная DB issue). Closes Sprint 1.1 part of #94. Frontend rendering этих 4 полей — отдельный PR (next: #112 / #115 / #114). * fix(site-finder): address PR #116 auto-review M1-M5 M1 (mutation risk): replace EMPTY_DUMP_RESULT direct refs with _make_empty_result() factory. dict(...) shallow copy left nested nspd_dump shared by reference across concurrent requests — single mutation pollutes module sentinel for all subsequent calls. Now каждый caller gets independent dict. M2 (O(N) spatial scan): SELECT extended denormalized counts (territorial_zones_count, zouit_count, engineering_count). Each spatial helper accepts layer_counts and early-returns when count=0 — skips heavy jsonb_array_elements + ST_Transform/ST_Intersects scan entirely. Critical для quarters с 2000+ features. M4 (documentation): _trigger_harvest docstring describes known burst/no-dedup limitation + TODO Redis SETNX (отдельный PR). M5 (test fragility): _make_db_mock_with_spatial docstring describes positional-call contract — db.execute order (0=dump, 1=zoning, 2=zouit, 3=engineering) и зависимость от count-values. +4 new tests (17 total, all pass): - test_make_empty_result_returns_independent_copies (mutation safety) - test_make_empty_result_overrides - test_early_exit_all_counts_zero_no_spatial_queries - test_early_exit_partial_counts Per auto-review on 3068a9c. * fix(site-finder): rename _make_empty_result → make_empty_result (public) per PR #116 review M1 residual fix: parcels.py exception path использовал EMPTY_DUMP_RESULT singleton ref вместо factory. Сейчас readonly access, но нарушает documented invariant модуля. Rename `_make_empty_result` → `make_empty_result` (public API), import в parcels.py, использовать в try/except fallback. Каждый request получает независимый dict — никаких shared references. M4 (Redis SETNX dedup) + M5 (test fragility) — deferred per review, documented в code/issue. Acceptable trade-offs: - M4: UPSERT idempotency делает данные safe; burst-duplicate task'и тратят WAF traffic впустую но не повреждают данные. - M5: docstring contractually describes positional-call order. 17/17 tests pass. ruff/format clean. Per auto-review on aef8308. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-13 09:14:19 +03:00
lekss361	9ee0b07003	feat(scrapers): search_by_quarter orchestrator + QuarterDump (#94 pt.2/4) (#109) ... Sprint 1.1 item #1 из плана #94 part 2. Foundation для PKK harvest pipeline — 1 vacuum (search) + N layer fetches → comprehensive snapshot всех NSPD данных в пределах квартала. Базис для G1 #28 ПЗЗ, G3 #30 ЗОУИТ, P2 #46 neighbors, E1 #51 parcels backfill, #96 ЕГРН помещения. Backend (nspd_client.py): - New QuarterDump frozen dataclass (slots=True): quarter + per-layer feature lists (parcels/buildings/territorial_zones/red_lines/engineering + zouit dict + risks dict) + bbox_3857 + layers_fetched (immutable tuple) + fetched_at_utc + total_features property. - New NSPDClient.search_by_quarter(quarter_cad, include_zouit=True, include_risks=False): search → bbox → bulk fetch per layer phase. Cost 6/11/22 requests. - New _geojson_bbox_3857() module-level helper — recursive coord walker. - Class constants QUARTER_CORE_LAYERS / QUARTER_ZOUIT_LAYERS / QUARTER_RISK_LAYERS. Empty-quarter (NSPD не нашёл cad): quarter=None, bbox=None, all lists empty, zouit/risks dicts populated с пустыми lists (структурно стабильно), layers_fetched=('search',). Tests: +12 tests (31 total, no network). Code review (code-reviewer pre-push): MINOR, fixed 3 of 5: - datetime import → module-level - layers_fetched → tuple[str, ...] (immutable in frozen dataclass) - docstring clarified empty-quarter semantics Bonus: ruff UP038 isinstance tuple → union syntax. Part of #94. Sprint 1.1: 4 PRs total. Next: migration → Celery → integration. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-12 18:11:30 +03:00
lekss361	bc092b58f1	feat(scrapers): NSPD client foundation (#94) — search + WMS + layers (#98) ... * feat(scrapers): NSPD client foundation (#94) — search_by_cad + WMS + layers Foundation для G1 #28 ПЗЗ, G2 #29 ВРИ, G3 #30 ЗОУИТ, E1 #51 cad_parcels, I3 #44 engineering, и поддержки on-demand #93. Backend (`app/services/scrapers/nspd_client.py`): - `NSPDClient` с 4 core методами: - `search_by_cad(cad, thematic_id)` — REST /api/geoportal/v2/search возвращает GeoJSON + ВРИ + land_category + cost_value за 1 запрос - `get_feature_info(layer_id, lon, lat, buffer_m=100)` — WMS GetFeatureInfo на точке: какие feature'ы layer'а покрывают (lon,lat) - `get_features_in_bbox(layer_id, bbox_3857)` — bulk через WMS workaround (WFS GetCapabilities → 404, поэтому большой bbox + центральная I/J точка) - `list_layers(theme_id)` — каталог слоёв в теме (PKK=1, ARN=665) - Typed responses: NSPDFeature, NSPDSearchResult, NSPDLayer (frozen dataclasses) - LAYERS catalog: 32 layer-id с семантическими именами (territorial_zones=875838, zouit_engineering=37578, risk_flooding=872205, etc) — TIER 1-6 per #94 - Coordinate helpers: lonlat_to_3857(), bbox_around_point_m() - Reuse: HEADERS + SSL ctx + fetch_geoportal из existing nspd_lite (WAF-compatible urllib trick). Rate limit через rate_ms. - WAF/Rate-limit: raises NspdLiteWafError на 403/429 — caller backoff. Tests (`backend/tests/test_nspd_client.py`): 14 unit tests, no network. - LAYERS catalog sanity (territorial_zones=875838 закрывает G1) - Coordinate transforms (zero, ЕКБ center, bbox) - NSPDFeature.from_raw parsing (full / missing fields) - NSPDSearchResult helpers (empty/first) - _walk_layer_tree (flat / nested / empty) - search_by_cad через monkeypatch fetch_geoportal Также включён micro-fix из PR #95 review (cosmetic): - parcels.py inline comment "до 25s" → "до 15s" (matched _INLINE_FETCH_WAIT_S) Scope NOT в этом PR (отдельные follow-up issues, sequential rule): - Schema migrations (cad_parcels_geom +columns, nspd_territorial_zones, nspd_zouit, nspd_red_lines, nspd_risk_zones tables) → отдельный PR - Celery tasks (sync_territorial_zones_bbox, sync_zouit_*, sync_risk_*) → отдельный PR использует client - Admin endpoints (trigger / status) → отдельный PR - Замена on-demand fetch на nspd_client → PR after schema ready Closes part of #94 (foundation only — sub-issues for schema/tasks). * fix(scrapers): address PR #98 auto-review minor feedback Backend (nspd_client.py): 1. (line 100+) bbox_around_point_m signature split на multi-line. 2. (lazy imports) import json, time → module-level. 3. (duplicate SSL ctx) Reuse _SSL_CTX из nspd_lite через explicit import. 4. (shape defense) list_layers чекает {"data": [...]} wrapper + non-dict/list garbage → warning + empty. 5. (cad_num docstring) search_by_cad документирует формат + ссылку на validate_cad_format. Tests: +5 mock тестов (19/19 passed): - get_feature_info URL builder + parsing - get_features_in_bbox bbox propagation - list_layers_walks_tree_response - list_layers_handles_data_wrapper - list_layers_handles_garbage_response Backend (parcels.py): docstring '~25с' → '~15с'. Per auto-review on be14388. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-12 09:47:45 +03:00
lekss361	49fac806fc	feat(site-finder): auto-fetch cadastre geometry on-demand (#93) (#95) ... * feat(site-finder): auto-fetch cadastre geometry on-demand (#93) Когда пользователь вводит cad-номер которого нет в БД (cad_quarters_geom / cad_buildings / cad_parcels_geom), вместо 404 «Загрузи через NSPD geo» (dead- end для non-admin) — теперь backend автоматически инициирует NSPD fetch, frontend показывает loading state с polling /fetch-status каждые 2с. Backend: - Новый модуль app/services/site_finder/cadastre_fetch.py с helpers: find_or_enqueue_fetch (atomic check + enqueue с дедупликацией по cad), fetch_status (smart polling endpoint — отличает not_in_nspd от failed), detect_thematic_id (3-сегм quarter / 4-сегм parcel / 5-сегм building), validate_cad_format. - Reuse: enqueue_geo_job + process_nspd_geo_job (workers/tasks/nspd_geo). source_kind='auto_on_demand' отличает от bulk; rate_ms=200, priority=9. - POST /parcels/{cad}/analyze graceful fallback: inline await до 25s (fast path), затем 202 + job_id + eta_seconds для polling, либо 400/404/503 в зависимости от статуса (с Retry-After 60s на 503). - GET /parcels/{cad}/fetch-status новый endpoint для polling. Frontend: - useSiteAnalysis расширен: fetchingState + cancel(). POST analyze + при 202 polling каждые 2с (max 60 итераций = 2 мин cap). status=ready → re-trigger analyze; not_in_nspd/failed/invalid_format → typed errors. - apiFetchWithStatus<T> + HTTPError в lib/api.ts — status-aware variant для 200 vs 202. - FetchingState.tsx: animated spinner, progress bar (linear до etaSeconds, asymptote после), elapsed counter, cancel button. Светло-голубая scheme отличается от обычного pending skeleton. - site-finder/page.tsx: FetchingState когда fetchingState активен; обычный pending skeleton — только при первичном analyze без 202. Edge cases (per #93 acceptance): - cad валидный в НСПД, fetch <25s → inline 200 OK - cad валидный, fetch >25s → 202 + frontend polls → ready → analyze - cad валидный, не в НСПД → 404 с понятным сообщением + формат hint - cad invalid format → 400 + формат hint - NSPD rate-limited / failed → 503 + Retry-After 60s - Параллельные запросы на тот же cad → один job, оба клиента poll'ят (дедуп через find_active_on_demand_job). Closes #93. * fix(site-finder): address PR #95 auto-review minor feedback Backend (cadastre_fetch.py): 1. (race condition) Advisory lock `pg_try_advisory_xact_lock(hashtext(cad_num))` обёрнут вокруг шагов "check active job → enqueue" в find_or_enqueue_fetch. Lock transaction-scoped, released at COMMIT. Параллельные запросы на тот же cad: первый получает lock и enqueue; второй lock=false → re-check active job (уже виден после первого COMMIT) → возвращает тот же job_id. Docstring обновлён, упоминание SELECT FOR UPDATE удалено. Backend (parcels.py): 3. (threadpool exhaustion) _INLINE_FETCH_WAIT_S снижен 25 → 15s с подробным комментарием: tradeoff про default Starlette anyio threadpool (40 slots) и concurrent burst saturation. 15s баланс: НСПД avg 5-15s для quarter, ~70% fast path; остальные 30% получают 202 + polling без блока. Data (87_on_demand_indexes.sql): 2. (missing index) New migration: - `nspd_geo_targets_cad_num_idx ON nspd_geo_targets(cad_num)` — для find_active/recent_completed_job (existing UNIQUE composite не покрывает WHERE cad_num=:c). - `nspd_geo_jobs_source_status_idx ON nspd_geo_jobs(source_kind, status)` composite для filter auto_on_demand + queued/running. Idempotent (CREATE INDEX IF NOT EXISTS), не блокер при текущем размере, критично при росте on-demand traffic. Frontend (useSiteAnalysis.ts): 4. (UI flicker) setFetchingState(null) перемещён ПОСЛЕ `await second apiFetch`. Иначе между clear и resolve есть момент когда mutation isPending=true + fetchingState=null → пустой экран ~1 RTT. NOT addressed (rebuttal): 5. (Tailwind convention) — проверил: в проекте нет ни globals.css, ни @tailwind directives. ВСЕ существующие site-finder components используют inline styles (ConfidenceBadge, GeotechRiskBlock, ScoreBreakdownPanel etc). Tailwind в deps но не wired up. Keep inline styles для consistency. 6. (animate-spin) — требует Tailwind globals (см. #5). `<style jsx>` keyframes — built-in Next.js, работает. Per auto-review on 2252236. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-12 09:02:17 +03:00
lekss361	49eadeb9ce	fix(pzz-sync): follow_redirects=True (PKK6 returns 301)	2026-05-11 22:50:46 +03:00
lekss361	1c1ecad8b8	fix(pzz-sync): disable SSL verify for Rosreestr PKK6 (self-signed cert chain)	2026-05-11 22:44:53 +03:00
lekss361	c31da62e8d	feat(analytics): recommend_mix v3.1-v3.4 - noise + 2D competitors + 24m cap + success-driven	2026-05-11 22:19:41 +03:00
lekss361	8be539ddc6	feat(site-finder): ПЗЗ territorial zones from Rosreestr PKK6 + zoning in analyze	2026-05-11 21:51:51 +03:00
lekss361	5b03d6d799	feat(site-finder): isochrones UI + networks VKH + OSM substations	2026-05-11 21:48:21 +03:00
lekss361	6ce634a9b9	feat(site-finder): utilities (power/pipeline) + fix Внешние факторы layout	2026-05-11 21:38:20 +03:00
lekss361	2855a01ca7	feat(site-finder): v3.5 - seasonal weather + hydrology + geotech risk	2026-05-11 21:32:07 +03:00
lekss361	7900dc5238	fix(site-finder): WKT LINESTRING needs commas + rosreestr_deals real column names	2026-05-11 20:58:49 +03:00
lekss361	1e9d32ee3c	feat(site-finder): v3.2 — noise + air quality + wind analytics	2026-05-11 20:42:37 +03:00
lekss361	f419900968	feat(site-finder): v3.1 — cad_parcels_geom, analyze fallback, POI lat/lon, OSM expand ... - nspd_geo: add _save_parcel() for thematic_id=1 → cad_parcels_geom (UPSERT, ST_Transform from Web Mercator); _persist_target now handles 1/2/5 - parcels.py: analyze endpoint geom lookup extended with cad_parcels_geom as 3rd fallback source (after cad_quarters_geom, cad_buildings); both SELECT and WKT subqueries updated - parcels.py: POI score_breakdown items now include lat/lon for map markers - poi_loader: OSM_CATEGORIES expanded — college+university→school, hypermarket→shop_supermarket; coverage +3 tag pairs	2026-05-11 19:52:19 +03:00
lekss361	3aeda297ee	fix(poi-sync): split Overpass query per category (504 Gateway Timeout)	2026-05-11 18:54:28 +03:00