c2e0428803
feat(nspd): get_feature_info FEATURE_COUNT=10 — multi-feature per point ( #1080 )
...
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m14s
Deploy / build-worker (push) Successful in 2m24s
Deploy / deploy (push) Successful in 1m21s
get_feature_info() шлёт WMS FEATURE_COUNT=10 (новый параметр feature_count, default 10). NSPD по умолчанию отдаёт 1 фичу/точку → перекрывающиеся зоны одного слоя теряются (37581 «иные ЗОУИТ» на ЕКБ = 2 фичи). Tile-геометрия не тронута. +2 теста (multi-feature parse, override), существующий обновлён.
Refs #1067 .
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-06 18:18:07 +00:00
38d948f078
fix(nspd): list_layers parse flat layers[] response, not children-tree ( #1077 )
...
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m26s
Deploy / deploy (push) Successful in 1m9s
Deploy / build-worker (push) Successful in 2m27s
NSPDClient.list_layers() парсил _walk_layer_tree по id/children, но НСПД layers-theme-tree отдаёт слои в плоском layers[] с ключом layerId → list_layers() молча возвращал [] для любой темы. Fix: сначала плоский data["layers"], иначе fallback на _walk_layer_tree; defensive None-guard на missing id. +тест на flat-форму, backward-compat сохранён.
Refs #1067 .
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-06 17:51:14 +00:00
e9585bcd3b
test(suite): re-align drifted tests with current code (CI-rehab 2/3)
...
Deploy / changes (push) Successful in 5s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 1m8s
Deploy / build-backend (push) Successful in 36s
Deploy / deploy (push) Successful in 1m13s
Suite was uncollectable for a while → code evolved, mocks didn't (42 failed /
5 errors → 1687 passed / 0 failed / 0 errors). TEST-ONLY, no app code changed,
no real bugs found (code-reviewed ✅ — no test weakened to fake-green).
- best_layouts: sum_deals→deals_window rename; velocity-scaling direction for
new SF-01 divisors (last_month=1.0/last_year=12.0)
- quarter_dump_lookup: 7→10-tuple mock rows (risks/opportunity/red_lines cols);
fix early-exit call-counts for the cad_zouit fallback (#232 )
- cadastre_bulk: add xmin/ymin/xmax/ymax to harvest-quarter db mock
- nspd_client: add required QuarterDump.opportunity
- TopLayoutRow: add required is_oversold
- analyze_{market_price,recent_permits,inline_weights}: SQL-signature dispatch
instead of fragile positional db.execute indices (SF-B5 query reorder; also
fixes a double-POST geom-starvation false-negative)
- custom_pois: move centroid >15km so center_bonus=0 isolates custom-POI delta
- layout/report PDF: runtime WeasyPrint probe (skip macOS dev, RUN on CI)
- mv_layout SQL: normalize +psycopg DSN + connectivity-probe skipif
- admin token tests: skip (X-Admin-Token gate removed #437/#426; protection is
Caddy basic_auth + RBAC, covered by test_rbac)
Refs #944 .
2026-06-03 19:05:43 +05:00
lekss361
9ee0b07003
feat(scrapers): search_by_quarter orchestrator + QuarterDump ( #94 pt.2/4) ( #109 )
...
Sprint 1.1 item #1 из плана #94 part 2. Foundation для PKK harvest pipeline —
1 vacuum (search) + N layer fetches → comprehensive snapshot всех NSPD данных
в пределах квартала. Базис для G1 #28 ПЗЗ, G3 #30 ЗОУИТ, P2 #46 neighbors,
E1 #51 parcels backfill, #96 ЕГРН помещения.
Backend (nspd_client.py):
- New QuarterDump frozen dataclass (slots=True): quarter + per-layer feature
lists (parcels/buildings/territorial_zones/red_lines/engineering + zouit
dict + risks dict) + bbox_3857 + layers_fetched (immutable tuple) +
fetched_at_utc + total_features property.
- New NSPDClient.search_by_quarter(quarter_cad, include_zouit=True,
include_risks=False): search → bbox → bulk fetch per layer phase.
Cost 6/11/22 requests.
- New _geojson_bbox_3857() module-level helper — recursive coord walker.
- Class constants QUARTER_CORE_LAYERS / QUARTER_ZOUIT_LAYERS /
QUARTER_RISK_LAYERS.
Empty-quarter (NSPD не нашёл cad): quarter=None, bbox=None, all lists empty,
zouit/risks dicts populated с пустыми lists (структурно стабильно),
layers_fetched=('search',).
Tests: +12 tests (31 total, no network).
Code review (code-reviewer pre-push): MINOR, fixed 3 of 5:
- datetime import → module-level
- layers_fetched → tuple[str, ...] (immutable in frozen dataclass)
- docstring clarified empty-quarter semantics
Bonus: ruff UP038 isinstance tuple → union syntax.
Part of #94 . Sprint 1.1: 4 PRs total. Next: migration → Celery → integration.
Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 18:11:30 +03:00
lekss361
bc092b58f1
feat(scrapers): NSPD client foundation ( #94 ) — search + WMS + layers ( #98 )
...
* feat(scrapers): NSPD client foundation (#94 ) — search_by_cad + WMS + layers
Foundation для G1 #28 ПЗЗ, G2 #29 ВРИ, G3 #30 ЗОУИТ, E1 #51 cad_parcels,
I3 #44 engineering, и поддержки on-demand #93 .
Backend (`app/services/scrapers/nspd_client.py`):
- `NSPDClient` с 4 core методами:
- `search_by_cad(cad, thematic_id)` — REST /api/geoportal/v2/search
возвращает GeoJSON + ВРИ + land_category + cost_value за 1 запрос
- `get_feature_info(layer_id, lon, lat, buffer_m=100)` — WMS GetFeatureInfo
на точке: какие feature'ы layer'а покрывают (lon,lat)
- `get_features_in_bbox(layer_id, bbox_3857)` — bulk через WMS workaround
(WFS GetCapabilities → 404, поэтому большой bbox + центральная I/J точка)
- `list_layers(theme_id)` — каталог слоёв в теме (PKK=1, ARN=665)
- Typed responses: NSPDFeature, NSPDSearchResult, NSPDLayer (frozen dataclasses)
- LAYERS catalog: 32 layer-id с семантическими именами (territorial_zones=875838,
zouit_engineering=37578, risk_flooding=872205, etc) — TIER 1-6 per #94
- Coordinate helpers: lonlat_to_3857(), bbox_around_point_m()
- Reuse: HEADERS + SSL ctx + fetch_geoportal из existing nspd_lite (WAF-compatible
urllib trick). Rate limit через rate_ms.
- WAF/Rate-limit: raises NspdLiteWafError на 403/429 — caller backoff.
Tests (`backend/tests/test_nspd_client.py`): 14 unit tests, no network.
- LAYERS catalog sanity (territorial_zones=875838 закрывает G1)
- Coordinate transforms (zero, ЕКБ center, bbox)
- NSPDFeature.from_raw parsing (full / missing fields)
- NSPDSearchResult helpers (empty/first)
- _walk_layer_tree (flat / nested / empty)
- search_by_cad через monkeypatch fetch_geoportal
Также включён micro-fix из PR #95 review (cosmetic):
- parcels.py inline comment "до 25s" → "до 15s" (matched _INLINE_FETCH_WAIT_S)
Scope NOT в этом PR (отдельные follow-up issues, sequential rule):
- Schema migrations (cad_parcels_geom +columns, nspd_territorial_zones,
nspd_zouit, nspd_red_lines, nspd_risk_zones tables) → отдельный PR
- Celery tasks (sync_territorial_zones_bbox, sync_zouit_*, sync_risk_*) →
отдельный PR использует client
- Admin endpoints (trigger / status) → отдельный PR
- Замена on-demand fetch на nspd_client → PR after schema ready
Closes part of #94 (foundation only — sub-issues for schema/tasks).
* fix(scrapers): address PR #98 auto-review minor feedback
Backend (nspd_client.py):
1. (line 100+) bbox_around_point_m signature split на multi-line.
2. (lazy imports) import json, time → module-level.
3. (duplicate SSL ctx) Reuse _SSL_CTX из nspd_lite через explicit import.
4. (shape defense) list_layers чекает {"data": [...]} wrapper + non-dict/list garbage → warning + empty.
5. (cad_num docstring) search_by_cad документирует формат + ссылку на validate_cad_format.
Tests: +5 mock тестов (19/19 passed):
- get_feature_info URL builder + parsing
- get_features_in_bbox bbox propagation
- list_layers_walks_tree_response
- list_layers_handles_data_wrapper
- list_layers_handles_garbage_response
Backend (parcels.py): docstring '~25с' → '~15с'.
Per auto-review on be14388
2026-05-12 09:47:45 +03:00
