9 commits

Author	SHA1	Message	Date
lekss361	b1530467f7	feat(sf): opportunity-ЗУ harvest → ird_overlays (future-supply, #1100) ... opportunity_harvest.py — grid-walk НСПД opportunity-слоёв (ЗУ аукцион/схема/свободные/межевание/ККР) по кварталам ЕКБ → ird_overlays с layer_kind='opportunity_*'. Переиспользует движок ird_harvest (#1090): _upsert_feature/_ekb_quarters. +krt_territories=37430 в LAYERS. Beat monthly (1-е 05:30). Self-contained, прод-прогон gated B6. Closes #1086. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 19:32:45 +00:00
lekss361	c2e0428803	feat(nspd): get_feature_info FEATURE_COUNT=10 — multi-feature per point (#1080) ... get_feature_info() шлёт WMS FEATURE_COUNT=10 (новый параметр feature_count, default 10). NSPD по умолчанию отдаёт 1 фичу/точку → перекрывающиеся зоны одного слоя теряются (37581 «иные ЗОУИТ» на ЕКБ = 2 фичи). Tile-геометрия не тронута. +2 теста (multi-feature parse, override), существующий обновлён. Refs #1067. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 18:18:07 +00:00
lekss361	38d948f078	fix(nspd): list_layers parse flat layers[] response, not children-tree (#1077) ... NSPDClient.list_layers() парсил _walk_layer_tree по id/children, но НСПД layers-theme-tree отдаёт слои в плоском layers[] с ключом layerId → list_layers() молча возвращал [] для любой темы. Fix: сначала плоский data["layers"], иначе fallback на _walk_layer_tree; defensive None-guard на missing id. +тест на flat-форму, backward-compat сохранён. Refs #1067. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 17:51:14 +00:00
lekss361	570c987e78	fix(#231): repair test_rate_limit_semaphore regression + docstring sync ... Re-review feedback (PR #260 head 8191a85): 1. tests/scrapers/test_nspd_bulk_client.py:360 — test referenced bulk_mod._SEMAPHORE which was renamed to _SEMAPHORE_LIMIT in previous commit. AttributeError на полном pytest run. Заменено: assert bulk_mod._SEMAPHORE_LIMIT == 3 (smoke). max_concurrent <= 3 уже валидирует throttling — capacity check был лишним. 2. app/services/scrapers/nspd_client.py docstrings (lines 483, 506) — обновлены ссылки NSPDBulkClient._SEMAPHORE(3) → NSPDBulkClient._sem (per-instance, capacity=3) чтобы соответствовать новой архитектуре.	2026-05-17 10:22:16 +03:00
lekss361	fdced94626	fix(#231): switch search_by_quarter area layers to grid-walk dispatch (Sub-PR B) ... - Add _GRID_WALK_LAYERS frozenset (territorial_zones, red_lines, engineering_structures, zouit_*, risk_*) - _fetch_layer dispatch: area layers → get_features_in_bbox_grid(grid_n=7, step_m=50) [foundation из PR #247], EGRN layers (parcels, buildings) → legacy get_features_in_bbox - Enrich engineering_structures features с classified_kind через classify_engineering_kind (PR #247 classifier) - Per-layer logger.info heartbeat (method + count) для прогресса - Bump harvest_quarter soft_time_limit 120 → 600s (grid-walk: 11 layers × 49 cells × ~70ms ≈ 40s + retries margin) - TestFetchLayerDispatch (3 tests): 34 passed, 1 skipped Pre-push deep-code-reviewer: APPROVE (1 medium nit — opportunity_oopt не в grid-walk, защищено через include_opportunity=False default; follow-up issue рекомендован).	2026-05-17 09:46:42 +03:00
lekss361	4519daf149	feat(#126): Sub-PR A — NSPDClient.get_features_in_bbox_grid + engineering classifier (#247)	2026-05-17 05:54:30 +00:00
lekss361	ed3c128528	feat(nspd): TIER 4 opportunity layers + red lines (#94 PR2 of 4) ... - NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt) + QuarterDump.opportunity field - nspd_sync: harvest_quarter accepts include_opportunity, denorm cols has_auction_parcels + opportunity_count in UPSERT - quarter_dump_lookup: - _get_opportunity_parcels (sort by distance, early-exit on count=0) - _get_red_lines (query existing dump.red_lines core path, layer='red_lines') - SQL 89_*: has_auction_parcels + opportunity_count + partial index - Pydantic: OpportunityParcel + RedLine schemas - /analyze: nspd_opportunity_parcels + nspd_red_lines fields - Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration - Tests: 28 total (11 PR1 + 17 PR2), all pass Red lines uses existing core harvest path (layer 879243 already in dump.red_lines from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed. Part of #94	2026-05-16 19:06:22 +03:00
lekss361	9ee0b07003	feat(scrapers): search_by_quarter orchestrator + QuarterDump (#94 pt.2/4) (#109) ... Sprint 1.1 item #1 из плана #94 part 2. Foundation для PKK harvest pipeline — 1 vacuum (search) + N layer fetches → comprehensive snapshot всех NSPD данных в пределах квартала. Базис для G1 #28 ПЗЗ, G3 #30 ЗОУИТ, P2 #46 neighbors, E1 #51 parcels backfill, #96 ЕГРН помещения. Backend (nspd_client.py): - New QuarterDump frozen dataclass (slots=True): quarter + per-layer feature lists (parcels/buildings/territorial_zones/red_lines/engineering + zouit dict + risks dict) + bbox_3857 + layers_fetched (immutable tuple) + fetched_at_utc + total_features property. - New NSPDClient.search_by_quarter(quarter_cad, include_zouit=True, include_risks=False): search → bbox → bulk fetch per layer phase. Cost 6/11/22 requests. - New _geojson_bbox_3857() module-level helper — recursive coord walker. - Class constants QUARTER_CORE_LAYERS / QUARTER_ZOUIT_LAYERS / QUARTER_RISK_LAYERS. Empty-quarter (NSPD не нашёл cad): quarter=None, bbox=None, all lists empty, zouit/risks dicts populated с пустыми lists (структурно стабильно), layers_fetched=('search',). Tests: +12 tests (31 total, no network). Code review (code-reviewer pre-push): MINOR, fixed 3 of 5: - datetime import → module-level - layers_fetched → tuple[str, ...] (immutable in frozen dataclass) - docstring clarified empty-quarter semantics Bonus: ruff UP038 isinstance tuple → union syntax. Part of #94. Sprint 1.1: 4 PRs total. Next: migration → Celery → integration. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-12 18:11:30 +03:00
lekss361	bc092b58f1	feat(scrapers): NSPD client foundation (#94) — search + WMS + layers (#98) ... * feat(scrapers): NSPD client foundation (#94) — search_by_cad + WMS + layers Foundation для G1 #28 ПЗЗ, G2 #29 ВРИ, G3 #30 ЗОУИТ, E1 #51 cad_parcels, I3 #44 engineering, и поддержки on-demand #93. Backend (`app/services/scrapers/nspd_client.py`): - `NSPDClient` с 4 core методами: - `search_by_cad(cad, thematic_id)` — REST /api/geoportal/v2/search возвращает GeoJSON + ВРИ + land_category + cost_value за 1 запрос - `get_feature_info(layer_id, lon, lat, buffer_m=100)` — WMS GetFeatureInfo на точке: какие feature'ы layer'а покрывают (lon,lat) - `get_features_in_bbox(layer_id, bbox_3857)` — bulk через WMS workaround (WFS GetCapabilities → 404, поэтому большой bbox + центральная I/J точка) - `list_layers(theme_id)` — каталог слоёв в теме (PKK=1, ARN=665) - Typed responses: NSPDFeature, NSPDSearchResult, NSPDLayer (frozen dataclasses) - LAYERS catalog: 32 layer-id с семантическими именами (territorial_zones=875838, zouit_engineering=37578, risk_flooding=872205, etc) — TIER 1-6 per #94 - Coordinate helpers: lonlat_to_3857(), bbox_around_point_m() - Reuse: HEADERS + SSL ctx + fetch_geoportal из existing nspd_lite (WAF-compatible urllib trick). Rate limit через rate_ms. - WAF/Rate-limit: raises NspdLiteWafError на 403/429 — caller backoff. Tests (`backend/tests/test_nspd_client.py`): 14 unit tests, no network. - LAYERS catalog sanity (territorial_zones=875838 закрывает G1) - Coordinate transforms (zero, ЕКБ center, bbox) - NSPDFeature.from_raw parsing (full / missing fields) - NSPDSearchResult helpers (empty/first) - _walk_layer_tree (flat / nested / empty) - search_by_cad через monkeypatch fetch_geoportal Также включён micro-fix из PR #95 review (cosmetic): - parcels.py inline comment "до 25s" → "до 15s" (matched _INLINE_FETCH_WAIT_S) Scope NOT в этом PR (отдельные follow-up issues, sequential rule): - Schema migrations (cad_parcels_geom +columns, nspd_territorial_zones, nspd_zouit, nspd_red_lines, nspd_risk_zones tables) → отдельный PR - Celery tasks (sync_territorial_zones_bbox, sync_zouit_*, sync_risk_*) → отдельный PR использует client - Admin endpoints (trigger / status) → отдельный PR - Замена on-demand fetch на nspd_client → PR after schema ready Closes part of #94 (foundation only — sub-issues for schema/tasks). * fix(scrapers): address PR #98 auto-review minor feedback Backend (nspd_client.py): 1. (line 100+) bbox_around_point_m signature split на multi-line. 2. (lazy imports) import json, time → module-level. 3. (duplicate SSL ctx) Reuse _SSL_CTX из nspd_lite через explicit import. 4. (shape defense) list_layers чекает {"data": [...]} wrapper + non-dict/list garbage → warning + empty. 5. (cad_num docstring) search_by_cad документирует формат + ссылку на validate_cad_format. Tests: +5 mock тестов (19/19 passed): - get_feature_info URL builder + parsing - get_features_in_bbox bbox propagation - list_layers_walks_tree_response - list_layers_handles_data_wrapper - list_layers_handles_garbage_response Backend (parcels.py): docstring '~25с' → '~15с'. Per auto-review on be14388. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-12 09:47:45 +03:00