lekss361
|
c8e510cbbe
|
feat(site-finder): weight profiles endpoints + analyze integration (#114 sub-PR 3/4)
Admin CRUD router + non-breaking analyze_parcel integration. 46 tests pass.
Refs: #114
|
2026-05-15 00:34:33 +03:00 |
|
lekss361
|
5aea78c2d8
|
feat(site-finder): weight profiles service — Pydantic + CRUD (#114 sub-PR 2/4) (#137)
Per #114 sub-PR 2/4. Pydantic v2 models + CRUD service. 14 passing tests.
API: list/get/get_default/create/update/delete_profile + resolve_weights
fallback to system defaults.
Validation: ALLOWED_CATEGORIES guard + weight bounds [-2.0, 3.0].
Default-uniqueness в одной transaction (job_settings.py pattern).
Vault: code/modules/Module_Weight_Profiles_Service.md.
Refs: #114
Co-authored-by: lekss361 <claudestars@proton.me>
|
2026-05-15 00:29:37 +03:00 |
|
lekss361
|
1c2f80a4b8
|
feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 FINAL) (#116)
* feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 #4 FINAL)
Замыкает Sprint 1.1 из #94 part 2 plan. После этого PR пользователь видит
свежие НСПД данные в UI (frontend integration — отдельный PR).
Backend (new app/services/site_finder/quarter_dump_lookup.py):
- `derive_quarter_cad(cad_num)` — 3/4/5-сегмент → quarter (3-segment)
- `get_quarter_dump_data(db, cad_num, parcel_wkt)` — main entrypoint:
- Reads nspd_quarter_dumps row для derived quarter
- Freshness threshold: 180 days
- Missing/stale/harvest_error → trigger harvest_quarter.apply_async() fire-
and-forget (lazy import против circular), return EMPTY_DUMP_RESULT
- Fresh + parcel_wkt=None → metadata only (no spatial queries)
- Fresh + geometry → 3 spatial queries via jsonb_array_elements + ST_Transform
(3857→4326) + ST_Intersects / ST_DWithin
- 3 private helpers:
- `_get_zoning` — point-in-polygon parcel centroid vs territorial_zones, LIMIT 1
- `_get_zouit_overlaps` — все zouit_% layers пересекающиеся с parcel
- `_get_engineering_nearby` — engineering_structures в 200m, sorted by distance
- `EMPTY_DUMP_RESULT` module-level constant — DRY для no-dump fallback (used
in get_quarter_dump_data internal + analyze_parcel try/except wrap)
Backend (parcels.py):
- Import EMPTY_DUMP_RESULT + get_quarter_dump_data
- Call wrapped в try/except — если nspd_quarter_dumps недоступна (DB timeout
/ table missing) → EMPTY_DUMP_RESULT fallback вместо 500 (consistent с
resilience pattern других optional fetches)
- Response gets 4 new fields:
- nspd_zoning: dict | None (G1 ПЗЗ — zone_code, zone_name, source)
- nspd_zouit_overlaps: list[dict] (G3 — overlaps в parcel, per ЗОУИТ group)
- nspd_engineering_nearby: list[dict] (I3 — engineering structures в 200m)
- nspd_dump: dict (freshness metadata — available, fetched_at_utc, stale,
harvest_triggered, total_features)
Tests: 13 new в test_quarter_dump_lookup.py (mock-based, no real DB):
- derive_quarter_cad 5 edge cases (3seg, 4seg, 5seg, invalid, whitespace)
- get_quarter_dump no_row → harvest triggered
- stale (>180d) → harvest triggered, stale=True
- harvest_error row → retry harvest triggered
- parcel_wkt=None → metadata only (1 DB call)
- fresh + zoning extraction
- fresh + zouit_overlaps list
- fresh integration: все 4 keys present
47 pre-existing tests still pass.
Code review (code-reviewer pre-push): MINOR, 0 blocking. Applied 2 of 4:
- ✅ #1: try/except wrap around get_quarter_dump_data в analyze_parcel
(защита от DB unavailability) + DRY через EMPTY_DUMP_RESULT module const
- ✅ #2: removed redundant nspd_zoning.fetched_at_utc (DRY — freshness в
nspd_dump.fetched_at_utc)
- ⏭ Deferred (acceptable): #3 ad-hoc harvest_quarter retry cooldown для
harvest_error rows (только при high traffic + persistent NSPD errors);
#4 raw_props в response — tech debt, убрать вместе с frontend PR
Performance note: 3 spatial queries per analyze adds ~10-50ms on typical
~100-feature quarter. Mitigation if quarters grow dense: materialized
per-layer sub-table (отдельная DB issue).
Closes Sprint 1.1 part of #94. Frontend rendering этих 4 полей — отдельный
PR (next: #112 / #115 / #114).
* fix(site-finder): address PR #116 auto-review M1-M5
M1 (mutation risk): replace EMPTY_DUMP_RESULT direct refs with
_make_empty_result() factory. dict(...) shallow copy left nested
nspd_dump shared by reference across concurrent requests — single
mutation pollutes module sentinel for all subsequent calls. Now
каждый caller gets independent dict.
M2 (O(N) spatial scan): SELECT extended denormalized counts
(territorial_zones_count, zouit_count, engineering_count). Each
spatial helper accepts layer_counts and early-returns when count=0
— skips heavy jsonb_array_elements + ST_Transform/ST_Intersects
scan entirely. Critical для quarters с 2000+ features.
M4 (documentation): _trigger_harvest docstring describes known
burst/no-dedup limitation + TODO Redis SETNX (отдельный PR).
M5 (test fragility): _make_db_mock_with_spatial docstring describes
positional-call contract — db.execute order (0=dump, 1=zoning, 2=zouit,
3=engineering) и зависимость от count-values.
+4 new tests (17 total, all pass):
- test_make_empty_result_returns_independent_copies (mutation safety)
- test_make_empty_result_overrides
- test_early_exit_all_counts_zero_no_spatial_queries
- test_early_exit_partial_counts
Per auto-review on 3068a9c.
* fix(site-finder): rename _make_empty_result → make_empty_result (public) per PR #116 review
M1 residual fix: parcels.py exception path использовал EMPTY_DUMP_RESULT
singleton ref вместо factory. Сейчас readonly access, но нарушает
documented invariant модуля.
Rename `_make_empty_result` → `make_empty_result` (public API), import в
parcels.py, использовать в try/except fallback. Каждый request получает
независимый dict — никаких shared references.
M4 (Redis SETNX dedup) + M5 (test fragility) — deferred per review,
documented в code/issue. Acceptable trade-offs:
- M4: UPSERT idempotency делает данные safe; burst-duplicate task'и тратят
WAF traffic впустую но не повреждают данные.
- M5: docstring contractually describes positional-call order.
17/17 tests pass. ruff/format clean.
Per auto-review on aef8308.
---------
Co-authored-by: lekss361 <claudestars@proton.me>
|
2026-05-13 09:14:19 +03:00 |
|
lekss361
|
f3b0ce5687
|
feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4 part A) (#111)
* feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4/4 part A)
Sprint 1.1 item #3 из #94 part 2. Замыкает harvest cycle: Celery task
оборачивает NSPDClient.search_by_quarter (PR #109) + UPSERT в
nspd_quarter_dumps (PR #110). После — analyze_parcel будет читать dumps
вместо on-demand NSPD HTTP в request-цикле (Sprint 1.1 #4, следующий PR).
Backend:
- New `backend/app/workers/tasks/nspd_sync.py`:
- `harvest_quarter(quarter_cad, region_code=66, include_zouit=True,
include_risks=False)` — single-quarter harvest. autoretry_for=
(NspdLiteWafError,) с retry_backoff=True, max_retries=3,
soft_time_limit=120s. Generic exception → upsert error row с
harvest_error=str(exc), counts=0 (НЕ re-raise).
- `harvest_stale_quarters(region_code, max_age_days=90, batch_size=50)` —
beat fanout: SELECT cad_quarters_geom EXCEPT fresh dumps → apply_async per cad.
- UPSERT использует ST_Multi(ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON,3857),4326))
для quarter_geom (schema MultiPolygon strict per database-expert contract).
- `celery_app.py`:
- Add `nspd_sync` в `include`
- Beat entry `nspd-harvest-stale-quarters` Mon 04:00 МСК, batch_size=50,
max_age_days=90
- `worker_ready` sanity check `SELECT 1 FROM nspd_quarter_dumps LIMIT 0`
(non-fatal critical log per Bug_Worker_Ready_EarlyReturn_Fixed pattern)
- `admin_scrape.py`:
- `HarvestQuarterRequest` Pydantic с pattern `^\d+:\d+:\d+$` (quarter only)
- `POST /api/v1/admin/scrape/nspd/harvest-quarter` manual trigger,
requires X-Admin-Token, returns task_id.
Tests (`test_nspd_sync.py`): 14 mock-based (no real NSPD/DB):
- happy path: search_by_quarter mock → verify SQL params/structure
- empty quarter → row with quarter_geom=NULL, total=0, no harvest_error
- WAF error → autoretry kicks in (raise re-propagated)
- Generic error → upsert error row
- Fanout: 50 stale → 50 apply_async calls
- Layer names preservation (all 5 zouit + 11 risks if enabled)
- Geometry SRID handling
Code review (code-reviewer pre-push): APPROVE, 0 blocking, 5 minor deferred:
- inline datetime import in error branch (style)
- layers_fetched array literal via string concat (current layer names safe)
- harvest_stale_quarters error rows always re-tried (intended, undocumented)
- test call_args.kwargs fragile (works for current callsite)
- beat collision with refresh-ekb-districts-medians every ~2 years (negligible)
Part of #94. Next: Sprint 1.1 #4 — analyze_parcel reads dump cache.
* fix(nspd-sync): address PR #111 auto-review M1-M5+M7 findings
M1 (runtime bug): remove WHERE region_code from outer cad_quarters_geom SELECT
in harvest_stale_quarters — column does not exist in prod schema
(cad_number, geom, raw_props, fetched_at, source). Would crash on first beat
tick. Inner nspd_quarter_dumps subquery keeps region_code filter unchanged.
M2 (correctness): _upsert_dump теперь делает db.rollback() в except перед
db.close() — prevents PendingRollbackError на следующем session use, если
PostGIS отбракует geometry или JSON cast failure.
M3 (low): worker_ready breadcrumb INSERT тоже теперь имеет rollback —
consistency с остальными session usages в celery_app.py.
M4 (style): layers_fetched — заменил manual string concat
'{' + ','.join(...) + '}' на native Python list. psycopg v3 сам сериализует
list → text[]. Защита от future layer names с запятыми/фигурными скобками.
M5 (style): import datetime as _dt поднят из else branch в top-level imports.
M7 (test cleanup): удалил dead __enter__/__exit__ mock setup в
test_harvest_stale_quarters_fanout / _empty. harvest_stale_quarters не
использует session как context manager — moc'и no-op.
14/14 tests pass. ruff/format clean.
Per auto-review on 9b2a289.
---------
Co-authored-by: lekss361 <claudestars@proton.me>
|
2026-05-12 18:45:57 +03:00 |
|
lekss361
|
9ee0b07003
|
feat(scrapers): search_by_quarter orchestrator + QuarterDump (#94 pt.2/4) (#109)
Sprint 1.1 item #1 из плана #94 part 2. Foundation для PKK harvest pipeline —
1 vacuum (search) + N layer fetches → comprehensive snapshot всех NSPD данных
в пределах квартала. Базис для G1 #28 ПЗЗ, G3 #30 ЗОУИТ, P2 #46 neighbors,
E1 #51 parcels backfill, #96 ЕГРН помещения.
Backend (nspd_client.py):
- New QuarterDump frozen dataclass (slots=True): quarter + per-layer feature
lists (parcels/buildings/territorial_zones/red_lines/engineering + zouit
dict + risks dict) + bbox_3857 + layers_fetched (immutable tuple) +
fetched_at_utc + total_features property.
- New NSPDClient.search_by_quarter(quarter_cad, include_zouit=True,
include_risks=False): search → bbox → bulk fetch per layer phase.
Cost 6/11/22 requests.
- New _geojson_bbox_3857() module-level helper — recursive coord walker.
- Class constants QUARTER_CORE_LAYERS / QUARTER_ZOUIT_LAYERS /
QUARTER_RISK_LAYERS.
Empty-quarter (NSPD не нашёл cad): quarter=None, bbox=None, all lists empty,
zouit/risks dicts populated с пустыми lists (структурно стабильно),
layers_fetched=('search',).
Tests: +12 tests (31 total, no network).
Code review (code-reviewer pre-push): MINOR, fixed 3 of 5:
- datetime import → module-level
- layers_fetched → tuple[str, ...] (immutable in frozen dataclass)
- docstring clarified empty-quarter semantics
Bonus: ruff UP038 isinstance tuple → union syntax.
Part of #94. Sprint 1.1: 4 PRs total. Next: migration → Celery → integration.
Co-authored-by: lekss361 <claudestars@proton.me>
|
2026-05-12 18:11:30 +03:00 |
|
lekss361
|
bc092b58f1
|
feat(scrapers): NSPD client foundation (#94) — search + WMS + layers (#98)
* feat(scrapers): NSPD client foundation (#94) — search_by_cad + WMS + layers
Foundation для G1 #28 ПЗЗ, G2 #29 ВРИ, G3 #30 ЗОУИТ, E1 #51 cad_parcels,
I3 #44 engineering, и поддержки on-demand #93.
Backend (`app/services/scrapers/nspd_client.py`):
- `NSPDClient` с 4 core методами:
- `search_by_cad(cad, thematic_id)` — REST /api/geoportal/v2/search
возвращает GeoJSON + ВРИ + land_category + cost_value за 1 запрос
- `get_feature_info(layer_id, lon, lat, buffer_m=100)` — WMS GetFeatureInfo
на точке: какие feature'ы layer'а покрывают (lon,lat)
- `get_features_in_bbox(layer_id, bbox_3857)` — bulk через WMS workaround
(WFS GetCapabilities → 404, поэтому большой bbox + центральная I/J точка)
- `list_layers(theme_id)` — каталог слоёв в теме (PKK=1, ARN=665)
- Typed responses: NSPDFeature, NSPDSearchResult, NSPDLayer (frozen dataclasses)
- LAYERS catalog: 32 layer-id с семантическими именами (territorial_zones=875838,
zouit_engineering=37578, risk_flooding=872205, etc) — TIER 1-6 per #94
- Coordinate helpers: lonlat_to_3857(), bbox_around_point_m()
- Reuse: HEADERS + SSL ctx + fetch_geoportal из existing nspd_lite (WAF-compatible
urllib trick). Rate limit через rate_ms.
- WAF/Rate-limit: raises NspdLiteWafError на 403/429 — caller backoff.
Tests (`backend/tests/test_nspd_client.py`): 14 unit tests, no network.
- LAYERS catalog sanity (territorial_zones=875838 закрывает G1)
- Coordinate transforms (zero, ЕКБ center, bbox)
- NSPDFeature.from_raw parsing (full / missing fields)
- NSPDSearchResult helpers (empty/first)
- _walk_layer_tree (flat / nested / empty)
- search_by_cad через monkeypatch fetch_geoportal
Также включён micro-fix из PR #95 review (cosmetic):
- parcels.py inline comment "до 25s" → "до 15s" (matched _INLINE_FETCH_WAIT_S)
Scope NOT в этом PR (отдельные follow-up issues, sequential rule):
- Schema migrations (cad_parcels_geom +columns, nspd_territorial_zones,
nspd_zouit, nspd_red_lines, nspd_risk_zones tables) → отдельный PR
- Celery tasks (sync_territorial_zones_bbox, sync_zouit_*, sync_risk_*) →
отдельный PR использует client
- Admin endpoints (trigger / status) → отдельный PR
- Замена on-demand fetch на nspd_client → PR after schema ready
Closes part of #94 (foundation only — sub-issues for schema/tasks).
* fix(scrapers): address PR #98 auto-review minor feedback
Backend (nspd_client.py):
1. (line 100+) bbox_around_point_m signature split на multi-line.
2. (lazy imports) import json, time → module-level.
3. (duplicate SSL ctx) Reuse _SSL_CTX из nspd_lite через explicit import.
4. (shape defense) list_layers чекает {"data": [...]} wrapper + non-dict/list garbage → warning + empty.
5. (cad_num docstring) search_by_cad документирует формат + ссылку на validate_cad_format.
Tests: +5 mock тестов (19/19 passed):
- get_feature_info URL builder + parsing
- get_features_in_bbox bbox propagation
- list_layers_walks_tree_response
- list_layers_handles_data_wrapper
- list_layers_handles_garbage_response
Backend (parcels.py): docstring '~25с' → '~15с'.
Per auto-review on be14388.
---------
Co-authored-by: lekss361 <claudestars@proton.me>
|
2026-05-12 09:47:45 +03:00 |
|
lekss361
|
b95559eac9
|
init
|
2026-04-25 13:45:19 +03:00 |
|