110 commits

Author	SHA1	Message	Date
lekss361	f160c5efaf	Merge pull request 'feat(#263 sub-A): cad_* overlay schemas + cadastre territorial_zones (ПЗЗ)' (#265) from feat/263-sub-a-cad-nspd-overlay-schemas-and-pzz into main	2026-05-17 07:50:08 +00:00
lekss361	8e377059a4	fix(#265 review): CRS 3857→4326 + stable zone_id hash + rename 102 + tx comment ... - #1 (BLOCK): ST_GeogFromGeoJSON → ST_Transform(ST_SetSRID(..., 3857), 4326)::geography so EPSG:3857 features from get_territorial_zones_in_bbox are correctly stored in GEOGRAPHY(POLYGON, 4326) column (matches cad_parcels/cad_buildings pattern) - #2: zone_id fallback changed from counter-based f"{quarter_cad}_{inserted}" to md5(json(props))[:12] — stable between runs, prevents wrong-row ON CONFLICT - #3: rename 101_cad_nspd_overlay_tables.sql → 102_* (101_user_custom_pois.sql exists) - #4: add comment clarifying SQLAlchemy autobegin guarantees outer tx before begin_nested() - Tests: updated test_feature_without_zone_id_uses_fallback to assert md5 format + idempotency	2026-05-17 10:45:52 +03:00
lekss361	f08db43145	feat(#263 sub-A): cad_* overlay schemas + cadastre territorial_zones (ПЗЗ) ... Phase 1 (migration 101): создаёт 5 новых таблиц cad_territorial_zones, cad_red_lines, cad_engineering_structures, cad_risk_zones, cad_opportunity_parcels. Idempotent (IF NOT EXISTS), GIST-индексы. Phase 2 partial (ПЗЗ only): - NSPDBulkClient.get_features_in_bbox_grid — async grid-walk WMS GetFeatureInfo - NSPDBulkClient.get_territorial_zones_in_bbox — convenience wrapper layer 875838 - harvest_quarter Phase 2.5 — вызов get_territorial_zones_in_bbox после grid-walk - _save_territorial_zones — UPSERT loop с SAVEPOINT per-row, ON CONFLICT zone_id 9 unit tests (mock DB): insert/upsert/no-geom/no-id/error-isolation/savepoint pattern.	2026-05-17 10:28:22 +03:00
lekss361	8ebcd15ed6	Merge pull request 'fix(#231): switch search_by_quarter area layers to grid-walk (Sub-PR B)' (#260) from fix/231-nspd-search-by-quarter-grid into main	2026-05-17 07:26:08 +00:00
lekss361	570c987e78	fix(#231): repair test_rate_limit_semaphore regression + docstring sync ... Re-review feedback (PR #260 head 8191a85): 1. tests/scrapers/test_nspd_bulk_client.py:360 — test referenced bulk_mod._SEMAPHORE which was renamed to _SEMAPHORE_LIMIT in previous commit. AttributeError на полном pytest run. Заменено: assert bulk_mod._SEMAPHORE_LIMIT == 3 (smoke). max_concurrent <= 3 уже валидирует throttling — capacity check был лишним. 2. app/services/scrapers/nspd_client.py docstrings (lines 483, 506) — обновлены ссылки NSPDBulkClient._SEMAPHORE(3) → NSPDBulkClient._sem (per-instance, capacity=3) чтобы соответствовать новой архитектуре.	2026-05-17 10:22:16 +03:00
lekss361	fdced94626	fix(#231): switch search_by_quarter area layers to grid-walk dispatch (Sub-PR B) ... - Add _GRID_WALK_LAYERS frozenset (territorial_zones, red_lines, engineering_structures, zouit_*, risk_*) - _fetch_layer dispatch: area layers → get_features_in_bbox_grid(grid_n=7, step_m=50) [foundation из PR #247], EGRN layers (parcels, buildings) → legacy get_features_in_bbox - Enrich engineering_structures features с classified_kind через classify_engineering_kind (PR #247 classifier) - Per-layer logger.info heartbeat (method + count) для прогресса - Bump harvest_quarter soft_time_limit 120 → 600s (grid-walk: 11 layers × 49 cells × ~70ms ≈ 40s + retries margin) - TestFetchLayerDispatch (3 tests): 34 passed, 1 skipped Pre-push deep-code-reviewer: APPROVE (1 medium nit — opportunity_oopt не в grid-walk, защищено через include_opportunity=False default; follow-up issue рекомендован).	2026-05-17 09:46:42 +03:00
lekss361	8cc94472f7	Merge pull request 'feat(#254): user_custom_pois — backend schema + CRUD + scoring integration' (#257) from feat/254-custom-pois-backend into main	2026-05-17 06:42:29 +00:00
lekss361	485f489bd2	feat(#254): user_custom_pois — backend schema + CRUD + scoring integration ... - SQL migration 101_user_custom_pois.sql: table with GEOGRAPHY(POINT) + GIST index - Pydantic schemas: CustomPoiCreate / CustomPoiUpdate / CustomPoiOut - Service custom_pois.py: CRUD + get_overlaps_for_scoring (1km radius, psycopg v3 CAST) - API /api/v1/custom-pois: POST(201)/GET/PATCH/DELETE, X-Session-Id header auth - Scoring in analyze_parcel: custom POI block after OSM POI loop, decay weight * max(0, 1 - dist/1000), custom_poi_score_items in response - Tests: CRUD + validation (weight/lon/lat 422) + scoring integration (mock-based) - Vault: Schema_User_Custom_Pois.md + Endpoint_Parcel_Analyze v3.9 section	2026-05-17 09:34:33 +03:00
lekss361	81cf274e80	fix(#234): address re-review nits — Settings singleton + docstring sync ... 1. _acquire_harvest_lock: Settings() instance → singleton 'settings' import from app.core.config — соответствует convention (scrape_kn.py, objective.py, celery_app.py все используют singleton). 2. _trigger_harvest docstring: убрана incorrect фраза «остальные ... видят harvest_eta_seconds от первого запроса» (код возвращает eta=None для denied-lock callers). Заменена на actual behavior: graceful no-data UI с ПКК fallback на FE. PR #250 re-review followup (issues #3 + docstring sync).	2026-05-17 09:31:43 +03:00
lekss361	b99ac76463	Merge remote-tracking branch 'forgejo/main' into fix/234-nspd-harvest-eta-badge ... # Conflicts: # backend/app/services/site_finder/quarter_dump_lookup.py	2026-05-17 09:19:12 +03:00
lekss361	990c109756	fix(#234): NSPD harvest ETA badge + SETNX dedupe (Sub-PR C) ... Backend (quarter_dump_lookup.py): - _acquire_harvest_lock: Redis SETNX TTL=120s на quarter, защищает от burst N concurrent analyze, ставящих N одинаковых harvest task в очередь - _trigger_harvest: использует lock перед apply_async, возвращает False если lock уже взят (другой запрос триггернул раньше) - make_empty_result/EMPTY_DUMP_RESULT: новое поле harvest_eta_seconds в nspd_dump dict, типичный harvest_quarter = 60с - /analyze: пробрасывает поле через nspd_dump dict (нет typed schema — response_model=None для /analyze endpoint, dict уходит как есть) Frontend (NspdFreshnessBadge, NspdZoningBlock): - Countdown «НСПД: загрузка ~Nс» вместо бесконечного спиннера - После остановки countdown (remaining=0) NspdZoningBlock показывает «загрузка дольше обычного» + ссылку на ПКК вместо infinite skeleton Tests: 5 новых unit + 2 для empty_result schema (всего +7, pass) Closes #234 (UX-side; data-side resolves когда Sub-PR B + D merged).	2026-05-17 09:13:06 +03:00
lekss361	10b79da2e9	fix(#243): cad_zouit fallback срабатывает когда nspd dump полностью отсутствует (#248)	2026-05-17 06:03:19 +00:00
lekss361	4a48d49474	fix(#242): ekburg_permits SSL — verify=False для CA Минцифры РФ	2026-05-17 08:57:42 +03:00
lekss361	4519daf149	feat(#126): Sub-PR A — NSPDClient.get_features_in_bbox_grid + engineering classifier (#247)	2026-05-17 05:54:30 +00:00
lekss361	f70be684da	feat(#29,#232): wire cad_parcels.permitted_use in analyze + cad_zouit fallback (G2+G3) ... TASK A (#29 G2): add parcel_meta to analyze response - New ParcelMeta Pydantic schema in app/schemas/parcel.py - SELECT from cad_parcels WHERE cad_num=:c in analyze_parcel() (step 9f) - Returns permitted_use_established_by_document, land_record_category_type, land_record_subtype, cost_value; None when row absent - Tests: test_analyze_parcel_meta.py (found + not-found cases) TASK B (#232 G3): cad_zouit fallback in _get_zouit_overlaps - When nspd_quarter_dumps has zouit_count==0, fall back to ST_Intersects query on cad_zouit (3483 rows, GIST indexed) - Overlaps tagged with source='cad_zouit'; format compatible with NSPD path - gate_verdict.py: BLOCKER_TYPE_ZONE_KEYWORDS tuple for keyword-based classification (охранная зона / трубопровод / электр / газ -> blocker; СЗЗ -> warning); NSPD subcategory path preserved backward-compat - Tests: 6 new test cases in test_gate_verdict.py covering cad_zouit path and backward-compat for NSPD subcategory path Updated db.execute call sequence in test_analyze_*.py (index shift +1 at pos 10).	2026-05-17 08:17:22 +03:00
lekss361	56bf28758c	Merge pull request 'feat(#115): Leaflet layer toggle для connection points (Макс KILLER)' (#230) from feat/connection-points-leaflet-layer into main	2026-05-16 20:00:55 +00:00
lekss361	a6e4ff0407	feat(#114): seed 3 default weight presets + include_system API param ... Add SQL migration 100_user_weight_profiles_default_seed.sql with system presets Эконом/Комфорт/Бизнес (user_id='__system__'). Migration is idempotent via ON CONFLICT DO UPDATE. Backend: - weight_profiles.py: add SYSTEM_USER_ID constant + list_profiles_with_system() - admin_weight_profiles.py: add include_system query param to GET list endpoint Tests: 3 new tests covering include_system flag and service sentinel behaviour.	2026-05-16 22:49:56 +03:00
lekss361	31581cedd2	fix(#112): remove broken status='sold' filter from competitors avg_price query ... domrf_kn_flats.status is NULL in ~99.8% of rows, so WHERE status='sold' always returned 0 rows and avg_price_per_m2 was always None. Drop the filter; AVG over all rows with price_per_m2 IS NOT NULL is semantically correct for a complex-level price estimate. Adds regression test test_competitors_avg_price_populated (Issue #227).	2026-05-16 22:46:10 +03:00
lekss361	db1ff5b1fc	feat(nspd): denorm nspd_parcels + nspd_buildings tables (#94 PR3 of 4) ... - data/sql/99_nspd_entities_denorm.sql: 2 tables (cad_num PK) + 6 indexes (quarter_cad, GIST geom, partial WHERE purpose ILIKE '%многокв%') - nspd_denorm.py: denorm_parcel_feature / denorm_building_feature / denorm_dump - SAVEPOINT per row via with db.begin_nested() - ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON(:geom), 3857), 4326) - json.dumps(geom, ensure_ascii=False) for psycopg bind - All CAST(:x AS type) — no ❌:type - nspd_sync.py: inline denorm after _upsert_dump in harvest_quarter (non-fatal) - nspd_denorm_backfill.py: Celery one-shot task (per-quarter commit) - admin_etl.py: POST /api/v1/admin/etl/nspd-denorm-backfill (AdminTokenAuth) - 19 tests: coerce helpers, denorm parcel/building, aggregate counts Foundation for downstream features (МКД neighbors lookup, parcel attrs). Part of #94	2026-05-16 20:20:03 +03:00
lekss361	5a7d405c77	Merge pull request 'feat(nspd): TIER 4 opportunity + red lines (#94 PR2 of 4)' (#220) from feat/94-pr2-opportunity-redlines into main	2026-05-16 17:09:52 +00:00
lekss361	fdb54834e5	fix(nspd): rename migration 89→98, fix red lines ST_Area→ST_Length (#220) ... - data/sql/98_*: rename from 89 to avoid collision with existing 89_drop_dead_brin - _get_red_lines: ST_Length(ST_Intersection(planar)::geography) instead of ST_Area(ST_Intersection(::geography, ::geography)) — fixes PostGIS 3.4 tolerance error and returns correct non-zero length for LINESTRING intersections - Rename intersection_area_sqm → intersection_length_m across schema, TS types, frontend component, and tests; add test_get_red_lines_db_exception_returns_empty Addresses review-bot feedback on PR #220.	2026-05-16 19:33:14 +03:00
lekss361	ed3c128528	feat(nspd): TIER 4 opportunity layers + red lines (#94 PR2 of 4) ... - NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt) + QuarterDump.opportunity field - nspd_sync: harvest_quarter accepts include_opportunity, denorm cols has_auction_parcels + opportunity_count in UPSERT - quarter_dump_lookup: - _get_opportunity_parcels (sort by distance, early-exit on count=0) - _get_red_lines (query existing dump.red_lines core path, layer='red_lines') - SQL 89_*: has_auction_parcels + opportunity_count + partial index - Pydantic: OpportunityParcel + RedLine schemas - /analyze: nspd_opportunity_parcels + nspd_red_lines fields - Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration - Tests: 28 total (11 PR1 + 17 PR2), all pass Red lines uses existing core harvest path (layer 879243 already in dump.red_lines from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed. Part of #94	2026-05-16 19:06:22 +03:00
lekss361	fbb76812f4	fix(recommend): cad_buildings.floors is INTEGER, not TEXT ... PROD 500 on /api/v1/analytics/recommend/mix: psycopg.errors.UndefinedFunction: operator does not exist: integer ~ unknown LINE 19: AND ((cb.floors ~ '^[0-9]+$' AND cb.f... _district_cadastre_baseline use regex \ on cb.floors. Schema is INTEGER (verified information_schema), so the regex defence against text values ('1-2', '2-3' etc) is misplaced — simplify to floors >= 3 with NULL safety via OR purpose ILIKE '%многокв%' fallback.	2026-05-16 18:29:35 +03:00
lekss361	1595157ebf	feat(nspd): enable TIER 3 risk zones harvest + extract in analyze (#94 PR1) ... - harvest_stale_quarters fanout passes include_risks=True to all queued tasks - _get_risk_zones() with ST_Intersects + ST_Area intersection calc - _extract_features_by_layer() generic helper for future PR2+ reuse - RiskZone Pydantic schema; /analyze returns nspd_risk_zones list - NspdRiskZonesBlock component with severity color-coding (red/yellow/orange) - 11 unit tests for extract/intersect/null-area/db-exception paths Part of #94 (TIER 3 risk layers — flooding, landslide, burns, erosion etc)	2026-05-16 16:11:59 +03:00
lekss361	dbf31cc911	feat(etl): objective_complex_mapping backfill via pg_trgm (#203) (#214)	2026-05-16 12:47:09 +00:00
lekss361	adc45f5724	feat(permits): ЕКБ РНС/РВЭ xlsx parser + schema + ETL (#105 PR A) (#211)	2026-05-16 11:41:21 +00:00
lekss361	efa66a9f68	feat(analytics): mv_quarter_price_per_m2 MV + refresh helper (#33 D1 PR A) (#209)	2026-05-16 11:12:01 +00:00
lekss361	092976f656	feat(site-finder): inline POI weights pass-through в /analyze (#201 Phase 1) (#206)	2026-05-16 11:00:41 +00:00
lekss361	256909d28b	feat(layouts): PDF ТЗ endpoint + BestLayoutsBlock UI (#113 PR D) (#199)	2026-05-16 09:53:05 +00:00
lekss361	ad16fc0e42	feat(parcels): best-layouts endpoint + service (#113 PR C) (#196)	2026-05-16 09:10:06 +00:00
lekss361	6aa22ef8ba	feat(analytics): mv_layout_velocity MV + refresh helper (#113 PR B) (#194)	2026-05-16 08:26:59 +00:00
lekss361	e673811a20	feat(parcels): layout signature schemas + extractor (#113 PR A) (#193)	2026-05-16 07:52:34 +00:00
lekss361	4deb1b0284	feat(parcels): connection-points endpoint (Forgejo #115 Phase 1 backend) (#190)	2026-05-16 05:58:27 +00:00
lekss361	e561df1b55	feat(parcels): competitors endpoint (Forgejo #112 Phase 1 backend) (#191)	2026-05-15 22:21:05 +00:00
lekss361	4d492ebc50	perf(cadastre): freshness skip + grid-walk heartbeat + log noise reduction (#187) ... * perf(cadastre): freshness skip + grid-walk heartbeat + reduce WMS log noise Three operational improvements after ekb_full v3 completed at 99.1%: Fix H — per-quarter freshness skip (scrape_cadastre.py): - enqueue_cadastre_harvest queries cad_quarter_stats.fetched_at per quarter - Skip quarters where fetched_at >= NOW() - skip_fresh_hours (default 1h) - extra_config.skip_fresh_hours=0 disables skip (full re-fetch) - Saves ~30k NSPD calls on repeated runs of same scope Fix I — heartbeat in grid-walk loop (bulk_harvest.py:_grid_walk_category): - 225 cells x 70-300ms = 15-75s heartbeat silence - Pass update_progress callback to _grid_walk_category - Call every heartbeat_every (50) cells with phase progress info - Prevents cleanup_zombies false-positive cancellation Fix J — NSPD WMS HTTP 500 noise to debug (bulk_harvest.py:_grid_walk_category): - NSPD WMS endpoint frequently returns HTTP 500 (server-side issue) - Was: logger.warning each, 200+ warnings per quarter - Now: logger.debug (visible via --log-level=DEBUG) 40 tests passing. * fix(cadastre): parse NSPD DD.MM.YYYY dates (Fix K — job 18 root cause) Job 18 (manual_list recovery 11 quarters) crashed every quarter with: psycopg.errors.DatetimeFieldOverflow: date/time field value out of range: '13.03.2023' NSPD returns registration_date in mixed formats: - DD.MM.YYYY for Сооружения / ОНС / ЕНК / ЗОУИТ - YYYY-MM-DD ISO for ЗУ / Здания - YYYY-MM-DD HH:MM:SS with time PG CAST(:date AS date) accepts ISO only — DD.MM.YYYY crashes. Add _parse_nspd_date helper: handles both formats + None/empty. Apply in upsert_construction, upsert_onc, upsert_enk, upsert_zouit, upsert_parcel for registration_date / build_record_registration_date / legal_act_date. 3 new unit tests for the helper (43 tests total). * fix(cadastre): address bot blockers — pivot Fix H to raw_targets + tests Bot review on PR #187 caught: 1. cadastre_jobs has no extra_config column → Fix H crashed on KeyError 2. Missing tests for Fix H + Fix I 3. _parse_nspd_date didn't validate date ranges Changes: - Pivot Fix H: read skip_fresh_hours from raw_targets JSONB (existing column), no schema migration needed - _parse_nspd_date: validate via date.fromisoformat() — rejects 2023-13-45 - Add 3 tests for Fix H in backend/tests/workers/test_scrape_cadastre.py - Add 2 tests for Fix I heartbeat callbacks in test_cadastre_bulk.py - Add 1 test for parse_nspd_date range validation (was: only format) 49 tests passing. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 23:02:37 +03:00
lekss361	d104222643	fix(cadastre,nspd_sync): SQL syntax bugs surfaced by ekb_full v2 (#186) ... Three distinct SQL syntax errors caught from worker logs: 1. **upsert_zouit** (bulk_harvest.py:940) — trailing comma after 4326 in ST_Transform(..., 3857), 4326,) → 'syntax error at or near ")"'. Hit on 8+ quarters in 0108 block (66:41:0108055/61/69/77 etc). Likely introduced by ruff-format auto-comma on closing paren. 2. **nspd_sync** (nspd_sync.py:120) — `:fetched_at_utc::timestamptz` psycopg double-colon trap → 'syntax error at or near ":"'. Crashed legacy on-demand harvest_quarter called from analyze. 3. **job_settings** (job_settings.py:224) — `:extra_config::jsonb` same double-colon trap in admin settings UPDATE. All converted to CAST(:x AS type) pattern (psycopg v3 safe). 40 tests still passing. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 19:18:43 +03:00
lekss361	5f8df655cc	fix(cadastre): cad_parcels.geom Polygon -> MultiPolygon (migration 93) (#185) ... NSPD returns MultiPolygon for Многоконтурный участок (e.g. 66:41:0105017:4) which crashed upsert_parcel — schema was strict POLYGON. Migration 93: ALTER COLUMN geom Polygon -> MultiPolygon USING ST_Multi(geom), DROP+RECREATE GIST index + back-compat VIEW cad_parcels_geom. upsert_parcel SQL wraps ST_Transform in ST_Multi() — coerces Polygon to MultiPolygon. NSPD returns both types (simple + multi-contour parcels). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 18:52:38 +03:00
lekss361	e3359c4f03	fix(cadastre): filter Point/LineString geometry in upsert_parcel/zouit (#184) ... cad_parcels.geom is geometry(Polygon, 4326) — strict schema. NSPD occasionally returns Point geometry for parcels without detailed boundary, causing INSERT failure: psycopg.errors.InvalidParameterValue: Geometry type (Point) does not match column type (Polygon) This killed pilot v8 at quarter 66:41:0104002 — worker autoretry exhausted, job hung at 25/50 (heartbeat stale). Fix: filter geometry.type at Python level in upsert_parcel + upsert_zouit (same pattern as upsert_quarter_geom_from_feature). Non-Polygon geometry → geom=NULL, raw_props preserved. Tables with permissive GEOMETRY schema (cad_buildings, cad_constructions, cad_enk, cad_oncs) unaffected — they accept any geometry type. Tests: 3 new (Point parcel → geom=NULL, Polygon happy path, LineString zouit). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 18:37:29 +03:00
lekss361	e695bca1e4	fix(cadastre): remove shared phase_state early-exit (race condition) (#183) ... cadastre_jobs.phase_state is shared between ALL parallel Celery workers of the same job. The early-exit triggered as soon as the FIRST worker wrote phase=done via progress_cb — all subsequent workers (reading same shared row) bailed without work. Effect: pilot v7 made 5 NSPD requests for 50 quarters (=0.1/quarter). Full ekb_full job #8: 4950 req / 2408 quarters = 2/quarter — 95% workers early-exited without doing snapshot phase. Root cause of 1.6% parcels coverage. Fix A/B/C only partially helped because most workers never reached Phase 1/1.5. Solution: delete the early-exit. Idempotency is guaranteed by ON CONFLICT DO UPDATE in every upsert_* — re-enqueued tasks write same rows without duplicates. Closes part of #168 (root-cause fix for race condition revealed by #182 metrics). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 17:41:27 +03:00
lekss361	057f8891dc	fix(cadastre): count snapshot+per-cat requests in cadastre_jobs.requests_count (#182) ... scrape_cadastre.py previously incremented requests_count only by grid_walk_requests, so snapshot-only quarters (and the entire ekb_full job after PR #179 skips grid-walk on broken geom) reported 0 NSPD requests — misleading for observability. Add HarvestResult.snapshot_requests counter (Phase 1 search + Phase 1.5 per-cat probes) and a total_requests property. Update Celery task to write total_requests into cadastre_jobs.requests_count. Tests added for the new property + Phase 1 snapshot_requests=1 assertion. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 17:17:01 +03:00
lekss361	905ef443fc	fix(cadastre): auto-heal cad_quarters_geom from CAT_QUARTER_STATS feature (#181) ... 72% of EKB quarters (1735/2408) have broken micro-precision geom in cad_quarters_geom. NSPD returns valid quarter polygon in CAT_QUARTER_STATS (36381) feature but Phase 4 was only consuming stats. Add upsert_quarter_geom_from_feature: CTE-based UPDATE that transforms 3857 polygon to 4326 and writes only when existing geom is NULL or broken (bbox width outside 100-10000m), verified via SQL-side sanity check. Wire into Phase 4 with begin_nested savepoint: malformed NSPD GeoJSON (self-intersecting ring) rollbacks only geom UPDATE, preserves upsert_quarter_stats. Closes part of #168 (follow-up data quality fix for #179). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 17:16:39 +03:00
lekss361	d5b57ccbf1	feat(cadastre): per-category NSPD snapshot — Phase 1.5 (ZOUIT/ENK) (#180) ... Switch base search_by_quarter from thematicSearchId=1 to thematic=1 (includes ZOUIT in features). Add optional category_id param that routes ZOUIT (36940/469039/469040/469042) -> thematicSearchId=5, ENK (39663) -> thematicSearchId=15. Add Phase 1.5 in harvest_quarter: for each PER_CATEGORY_PROBE_CATS category present in meta_counts (meta_total>0), run dedicated per-category search call. Skips if category in GRID_WALK_CATS and bbox is valid (grid-walk gives better coverage). Curl-probe confirmed NSPD API caps at 20/category with no pagination; categoryId param is server-ignored; thematic=1 strictly better than thematicSearchId=1 (includes ZOUIT). Closes part of #168 (ZOUIT/ENK coverage gap). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 16:53:49 +03:00
lekss361	3beb75d68a	fix(cadastre): skip grid-walk for broken quarter geometries (#168) (#179) ... * fix(cadastre): skip grid-walk for broken quarter geometries (#168) Pilot v5 (50 quarters) разоблачил что cad_quarters_geom имеет 1735/2408 (72%) ЕКБ rows с broken micro-precision geom (width ~0.01м instead of 200-4000м). Causes: - grid-walk WMS receives bbox ~1mm × 1mm → 500 ServiceException на all 450 req - One quarter hung 2+ min (silent failure in async loop) Real fix: re-import cad_quarters_geom geometry from NSPD (category 36381). Tracked в follow-up issue. Quick fix (this PR): sanity-check quarter_bbox_3857 — return None если width ИЛИ height not in [100m, 10000m]. Bulk_harvest_quarter then skips grid-walk для broken quarters, только snapshot phase запускается (20 per cat). After this fix: - All 50 pilot quarters получают snapshot (parcels/buildings/etc up to 20 each) - Grid-walk runs only для 673/2408 healthy quarters → no 500 cascades, no hangs * test(cadastre): add quarter_bbox_3857 boundary tests (PR #179 bot blocker) - healthy bbox 750m -> returns tuple - width <100m (broken micro-precision) -> None - width >10000m (oversized) -> None - height <100m (symmetry) -> None - row missing (quarter not in cad_quarters_geom) -> None --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 16:25:06 +03:00
lekss361	6dfff5e046	fix(cadastre): CAST :geom AS text — psycopg AmbiguousParameter (#168) (#178) ... Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 16:02:03 +03:00
lekss361	5da66bbc93	feat(cadastre): bulk_harvest worker + grid-walker + admin API (#168 PR3/5) (#171) ... * feat(cadastre): bulk_harvest_quarter Celery task + grid-walker + saga state (#168 PR3/5) Add bulk cadastre harvest pipeline: - services/cadastre/bulk_harvest.py: async harvest_quarter() orchestrator (4 phases) + 7 upsert helpers (parcels/buildings/constructions/oncs/enks/zouit/quarter_stats) using CAST(:x AS jsonb) pattern, begin_nested() SAVEPOINT per grid-walk upsert - services/cadastre/grid_geometry.py: quarter_bbox_3857 (PostGIS ST_Extent) + generate_grid_click_points (15x15 = 225 sub-bbox grid) - workers/tasks/scrape_cadastre.py: bulk_harvest_quarter_task (acks_late=True, dont_autoretry_for NspdBulkWafError), enqueue_cadastre_harvest, cleanup_zombies - api/v1/admin_cadastre.py: 5 endpoints behind AdminTokenAuth — create/list/get/cancel/resume - Tests: 13 service unit + 8 API tests * fixup(cadastre): drop importorskip (PR2 merged) — imports now top-level (#168 PR3) * fixup(cadastre): tile_size unified param + register slow marker (#168 PR3) Blocker #1: rename tile_width/tile_height → tile_size in generate_grid_click_points. Callers in bulk_harvest.py and test_cadastre_bulk.py used tile_size; def had tile_width+tile_height → TypeError at runtime. Blocker #2 (10 failures in test_admin_cadastre.py) was side-effect of #1: TypeError at import chain (app.main → admin_cadastre → bulk_harvest) broke FastAPI app load → dependency_overrides AttributeError. Now resolved. Also register `slow` pytest marker in pyproject.toml to suppress PytestUnknownMarkWarning. * fixup(cadastre): use importlib.util.find_spec instead of import (#168 PR3) Root cause Blocker #2: `import app.workers.tasks.scrape_cadastre` at module top-level REBOUND `app` from FastAPI instance (line 14) to the Python package. All subsequent `app.dependency_overrides[get_db] = ...` failed with AttributeError because `app` was now the namespace module. Fix: probe module availability with importlib.util.find_spec — does not import or rebind names. FastAPI `app` instance stays intact. All 28 cadastre tests pass locally. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 13:31:32 +03:00
lekss361	d7fbaa0528	feat(velocity): per-room-bucket breakdown in VelocityResult (#163) ... * fix(velocity): :window_interval::interval cast syntax — same bug as PR #152 SQLAlchemy text() parser confuses ':window_interval' (named param) followed by '::interval' (cast operator) — exactly как было в PR #152 с :weights::jsonb. Result: psycopg видит literal ':window_interval' в SQL → syntax error → exception caught by velocity catch → return None → UI shows null. ## Fix ':window_interval::interval' → 'CAST(:window_interval AS interval)' (2 мест: sales_rows query + _get_ekb_median percentile). Pre-push code-reviewer должен был catch — добавим в feedback что для каждого text() SQL grep ':[a-z]*::' before push. Refs: PR #158 deploy verify, e2e velocity = null root cause * feat(velocity): add per-room-bucket breakdown to VelocityResult Add third SQL query (bucket_rows) aggregating deals_total_count by room_bucket ('студия','1','2','3','4+') across mapped competitors. New fields: - VelocityResult.by_room_bucket: aggregate {units, sqm, complexes_count} - sample_competitors[].by_room_bucket: {bucket: units} Bucket query wrapped в SAVEPOINT — failure degrades gracefully к empty dict без abort outer tx. CAST(:window_interval AS interval) pattern per PR #160 lesson. Test coverage: 3 new tests (aggregation, empty fallback, sample entries). Closes part of #161 --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 10:09:03 +03:00
lekss361	46584d7c75	fix(velocity): :window_interval::interval cast syntax — same bug as PR #152 (#160) ... SQLAlchemy text() parser confuses ':window_interval' (named param) followed by '::interval' (cast operator) — exactly как было в PR #152 с :weights::jsonb. Result: psycopg видит literal ':window_interval' в SQL → syntax error → exception caught by velocity catch → return None → UI shows null. ## Fix ':window_interval::interval' → 'CAST(:window_interval AS interval)' (2 мест: sales_rows query + _get_ekb_median percentile). Pre-push code-reviewer должен был catch — добавим в feedback что для каждого text() SQL grep ':[a-z]*::' before push. Refs: PR #158 deploy verify, e2e velocity = null root cause Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 09:46:01 +03:00
lekss361	8fa1d005fd	feat(velocity): migrate D2 sales source → objective_corpus_room_month (#158) ... * feat(velocity): migrate D2 sales source from stale sale_graph to objective_corpus_room_month Closes #156 * fix(velocity): cross-stack contract + NULL safety per pre-push review 3 pre-push code-reviewer findings fixed: 1. TS union extended ('objective'|'sale_graph'|'rosreestr_fallback') + UI conditions handle both objective и sale_graph как valid sources. 2. COALESCE(deals_total_vol_m2, deals_total_count * 45.0) — NULL safety for DKP-only rows (vol_m2 nullable, count > 0). 3. room_bucket parking filter — verified false positive (все 5 buckets apartments: студия/1/2/3/4+). Refs: PR #157 pre-push code-reviewer * fix(types): add 'objective' to velocity_source union (cherry-pick miss) Cherry-pick от закрытого PR #157 потерял этот файл. Frontend type-check fails с 'no overlap' error. Restoring. Refs: PR #158 frontend CI failure --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 09:34:00 +03:00
lekss361	c39b53cf53	fix(site-finder): cascade InFailedSqlTransaction — velocity internal rollback (#155) ... * fix(site-finder): db.rollback() в velocity catches + SAVEPOINT zoning/success_rec velocity.py 3 internal try/except возвращали None БЕЗ db.rollback(). PR #154 SAVEPOINT не помог — exception не propagates из velocity. Postgres tx aborted внутри, cascade InFailedSqlTransaction на следующих queries. Fix: - velocity.py: db.rollback() в 3 catch блоках - parcels.py: SAVEPOINT обёртки для zoning + success_recommendation Refs: user reports 2026-05-15 cascade 500 * fix(velocity): SAVEPOINT per query (not db.rollback) — correct SQLAlchemy 2.0 pattern Per PR #155 bot review — мой db.rollback() в catches был НЕПРАВИЛЬНЫЙ. Session.rollback() = ENTIRE outer tx (не savepoint). Outer SAVEPOINT context становится orphaned → InvalidRequestError на __exit__. Correct fix: wrap каждый db.execute в свой with db.begin_nested(): - Failure → __exit__ propagates exception → SAVEPOINT rolls back - Outer tx остаётся clean - velocity returns None gracefully → caller продолжает Все 3 SQL queries в velocity.py теперь SAVEPOINT-wrapped: - compute_velocity: competitor_query - compute_velocity: sales_rows query - _get_ekb_median: median query db.rollback() removed из всех 3 catches. Refs: PR #155 bot review --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 08:52:38 +03:00
lekss361	af39e95f83	fix(site-finder): SQLAlchemy syntax error :weights::jsonb → CAST(:weights AS jsonb) (#152) ... Production POST /weight-profiles → 500 SyntaxError. ## Root cause SQLAlchemy text() parser трактует ':weights' followed by '::jsonb' двусмысленно: - ':weights' — named param binding (должен стать %(weights)s) - '::jsonb' — PG cast operator Parser НЕ распознаёт :weights как параметр когда сразу следует :: cast. Result: psycopg видит literal ':weights' в SQL → syntax error. ``` LINE 5: ($1, $2, :weights::jsonb, $3, $4) ^ parameters: {user_id, profile_name, is_default, description} ↑ note: 'weights' missing — SQLAlchemy skipped it ``` ## Fix Replace ambiguous :weights::jsonb с CAST(:weights AS jsonb): - _INSERT (line 102) - _UPDATE dynamic 'weights = :weights::jsonb' (line 273) CAST() syntax не имеет ambiguity, SQLAlchemy корректно bind'ит :weights. ## Affected endpoints - POST /api/v1/admin/site-finder/weight-profiles (create) - PUT /api/v1/admin/site-finder/weight-profiles/{id} (update with weights) Refs: user report 2026-05-15 Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 08:12:18 +03:00