Commit graph

15 commits

Author SHA1 Message Date
lekss361
1c2f80a4b8
feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 FINAL) (#116)
* feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 #4 FINAL)

Замыкает Sprint 1.1 из #94 part 2 plan. После этого PR пользователь видит
свежие НСПД данные в UI (frontend integration — отдельный PR).

Backend (new app/services/site_finder/quarter_dump_lookup.py):
- `derive_quarter_cad(cad_num)` — 3/4/5-сегмент → quarter (3-segment)
- `get_quarter_dump_data(db, cad_num, parcel_wkt)` — main entrypoint:
  - Reads nspd_quarter_dumps row для derived quarter
  - Freshness threshold: 180 days
  - Missing/stale/harvest_error → trigger harvest_quarter.apply_async() fire-
    and-forget (lazy import против circular), return EMPTY_DUMP_RESULT
  - Fresh + parcel_wkt=None → metadata only (no spatial queries)
  - Fresh + geometry → 3 spatial queries via jsonb_array_elements + ST_Transform
    (3857→4326) + ST_Intersects / ST_DWithin
- 3 private helpers:
  - `_get_zoning` — point-in-polygon parcel centroid vs territorial_zones, LIMIT 1
  - `_get_zouit_overlaps` — все zouit_% layers пересекающиеся с parcel
  - `_get_engineering_nearby` — engineering_structures в 200m, sorted by distance
- `EMPTY_DUMP_RESULT` module-level constant — DRY для no-dump fallback (used
  in get_quarter_dump_data internal + analyze_parcel try/except wrap)

Backend (parcels.py):
- Import EMPTY_DUMP_RESULT + get_quarter_dump_data
- Call wrapped в try/except — если nspd_quarter_dumps недоступна (DB timeout
  / table missing) → EMPTY_DUMP_RESULT fallback вместо 500 (consistent с
  resilience pattern других optional fetches)
- Response gets 4 new fields:
  - nspd_zoning: dict | None (G1 ПЗЗ — zone_code, zone_name, source)
  - nspd_zouit_overlaps: list[dict] (G3 — overlaps в parcel, per ЗОУИТ group)
  - nspd_engineering_nearby: list[dict] (I3 — engineering structures в 200m)
  - nspd_dump: dict (freshness metadata — available, fetched_at_utc, stale,
    harvest_triggered, total_features)

Tests: 13 new в test_quarter_dump_lookup.py (mock-based, no real DB):
- derive_quarter_cad 5 edge cases (3seg, 4seg, 5seg, invalid, whitespace)
- get_quarter_dump no_row → harvest triggered
- stale (>180d) → harvest triggered, stale=True
- harvest_error row → retry harvest triggered
- parcel_wkt=None → metadata only (1 DB call)
- fresh + zoning extraction
- fresh + zouit_overlaps list
- fresh integration: все 4 keys present

47 pre-existing tests still pass.

Code review (code-reviewer pre-push): MINOR, 0 blocking. Applied 2 of 4:
-  #1: try/except wrap around get_quarter_dump_data в analyze_parcel
  (защита от DB unavailability) + DRY через EMPTY_DUMP_RESULT module const
-  #2: removed redundant nspd_zoning.fetched_at_utc (DRY — freshness в
  nspd_dump.fetched_at_utc)
- ⏭ Deferred (acceptable): #3 ad-hoc harvest_quarter retry cooldown для
  harvest_error rows (только при high traffic + persistent NSPD errors);
  #4 raw_props в response — tech debt, убрать вместе с frontend PR

Performance note: 3 spatial queries per analyze adds ~10-50ms on typical
~100-feature quarter. Mitigation if quarters grow dense: materialized
per-layer sub-table (отдельная DB issue).

Closes Sprint 1.1 part of #94. Frontend rendering этих 4 полей — отдельный
PR (next: #112 / #115 / #114).

* fix(site-finder): address PR #116 auto-review M1-M5

M1 (mutation risk): replace EMPTY_DUMP_RESULT direct refs with
_make_empty_result() factory. dict(...) shallow copy left nested
nspd_dump shared by reference across concurrent requests — single
mutation pollutes module sentinel for all subsequent calls. Now
каждый caller gets independent dict.

M2 (O(N) spatial scan): SELECT extended denormalized counts
(territorial_zones_count, zouit_count, engineering_count). Each
spatial helper accepts layer_counts and early-returns when count=0
— skips heavy jsonb_array_elements + ST_Transform/ST_Intersects
scan entirely. Critical для quarters с 2000+ features.

M4 (documentation): _trigger_harvest docstring describes known
burst/no-dedup limitation + TODO Redis SETNX (отдельный PR).

M5 (test fragility): _make_db_mock_with_spatial docstring describes
positional-call contract — db.execute order (0=dump, 1=zoning, 2=zouit,
3=engineering) и зависимость от count-values.

+4 new tests (17 total, all pass):
- test_make_empty_result_returns_independent_copies (mutation safety)
- test_make_empty_result_overrides
- test_early_exit_all_counts_zero_no_spatial_queries
- test_early_exit_partial_counts

Per auto-review on 3068a9c.

* fix(site-finder): rename _make_empty_result → make_empty_result (public) per PR #116 review

M1 residual fix: parcels.py exception path использовал EMPTY_DUMP_RESULT
singleton ref вместо factory. Сейчас readonly access, но нарушает
documented invariant модуля.

Rename `_make_empty_result` → `make_empty_result` (public API), import в
parcels.py, использовать в try/except fallback. Каждый request получает
независимый dict — никаких shared references.

M4 (Redis SETNX dedup) + M5 (test fragility) — deferred per review,
documented в code/issue. Acceptable trade-offs:
- M4: UPSERT idempotency делает данные safe; burst-duplicate task'и тратят
  WAF traffic впустую но не повреждают данные.
- M5: docstring contractually describes positional-call order.

17/17 tests pass. ruff/format clean.

Per auto-review on aef8308.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-13 09:14:19 +03:00
lekss361
49fac806fc
feat(site-finder): auto-fetch cadastre geometry on-demand (#93) (#95)
* feat(site-finder): auto-fetch cadastre geometry on-demand (#93)

Когда пользователь вводит cad-номер которого нет в БД (cad_quarters_geom /
cad_buildings / cad_parcels_geom), вместо 404 «Загрузи через NSPD geo» (dead-
end для non-admin) — теперь backend автоматически инициирует NSPD fetch,
frontend показывает loading state с polling /fetch-status каждые 2с.

Backend:
- Новый модуль app/services/site_finder/cadastre_fetch.py с helpers:
  find_or_enqueue_fetch (atomic check + enqueue с дедупликацией по cad),
  fetch_status (smart polling endpoint — отличает not_in_nspd от failed),
  detect_thematic_id (3-сегм quarter / 4-сегм parcel / 5-сегм building),
  validate_cad_format.
- Reuse: enqueue_geo_job + process_nspd_geo_job (workers/tasks/nspd_geo).
  source_kind='auto_on_demand' отличает от bulk; rate_ms=200, priority=9.
- POST /parcels/{cad}/analyze graceful fallback: inline await до 25s
  (fast path), затем 202 + job_id + eta_seconds для polling, либо
  400/404/503 в зависимости от статуса (с Retry-After 60s на 503).
- GET /parcels/{cad}/fetch-status новый endpoint для polling.

Frontend:
- useSiteAnalysis расширен: fetchingState + cancel(). POST analyze + при 202
  polling каждые 2с (max 60 итераций = 2 мин cap). status=ready → re-trigger
  analyze; not_in_nspd/failed/invalid_format → typed errors.
- apiFetchWithStatus<T> + HTTPError в lib/api.ts — status-aware variant
  для 200 vs 202.
- FetchingState.tsx: animated spinner, progress bar (linear до etaSeconds,
  asymptote после), elapsed counter, cancel button. Светло-голубая
  scheme отличается от обычного pending skeleton.
- site-finder/page.tsx: FetchingState когда fetchingState активен; обычный
  pending skeleton — только при первичном analyze без 202.

Edge cases (per #93 acceptance):
- cad валидный в НСПД, fetch <25s → inline 200 OK
- cad валидный, fetch >25s → 202 + frontend polls → ready → analyze
- cad валидный, не в НСПД → 404 с понятным сообщением + формат hint
- cad invalid format → 400 + формат hint
- NSPD rate-limited / failed → 503 + Retry-After 60s
- Параллельные запросы на тот же cad → один job, оба клиента poll'ят (дедуп
  через find_active_on_demand_job).

Closes #93.

* fix(site-finder): address PR #95 auto-review minor feedback

Backend (cadastre_fetch.py):
1. (race condition) Advisory lock `pg_try_advisory_xact_lock(hashtext(cad_num))`
   обёрнут вокруг шагов "check active job → enqueue" в find_or_enqueue_fetch.
   Lock transaction-scoped, released at COMMIT. Параллельные запросы на тот
   же cad: первый получает lock и enqueue; второй lock=false → re-check
   active job (уже виден после первого COMMIT) → возвращает тот же job_id.
   Docstring обновлён, упоминание SELECT FOR UPDATE удалено.

Backend (parcels.py):
3. (threadpool exhaustion) _INLINE_FETCH_WAIT_S снижен 25 → 15s с подробным
   комментарием: tradeoff про default Starlette anyio threadpool (40 slots)
   и concurrent burst saturation. 15s баланс: НСПД avg 5-15s для quarter,
   ~70% fast path; остальные 30% получают 202 + polling без блока.

Data (87_on_demand_indexes.sql):
2. (missing index) New migration:
   - `nspd_geo_targets_cad_num_idx ON nspd_geo_targets(cad_num)` — для
     find_active/recent_completed_job (existing UNIQUE composite не покрывает
     WHERE cad_num=:c).
   - `nspd_geo_jobs_source_status_idx ON nspd_geo_jobs(source_kind, status)`
     composite для filter auto_on_demand + queued/running.
   Idempotent (CREATE INDEX IF NOT EXISTS), не блокер при текущем размере,
   критично при росте on-demand traffic.

Frontend (useSiteAnalysis.ts):
4. (UI flicker) setFetchingState(null) перемещён ПОСЛЕ `await second
   apiFetch`. Иначе между clear и resolve есть момент когда mutation
   isPending=true + fetchingState=null → пустой экран ~1 RTT.

NOT addressed (rebuttal):
5. (Tailwind convention) — проверил: в проекте нет ни globals.css, ни
   @tailwind directives. ВСЕ существующие site-finder components используют
   inline styles (ConfidenceBadge, GeotechRiskBlock, ScoreBreakdownPanel etc).
   Tailwind в deps но не wired up. Keep inline styles для consistency.
6. (animate-spin) — требует Tailwind globals (см. #5). `<style jsx>`
   keyframes — built-in Next.js, работает.

Per auto-review on 2252236.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 09:02:17 +03:00
lekss361
49eadeb9ce fix(pzz-sync): follow_redirects=True (PKK6 returns 301) 2026-05-11 22:50:46 +03:00
lekss361
1c1ecad8b8 fix(pzz-sync): disable SSL verify for Rosreestr PKK6 (self-signed cert chain) 2026-05-11 22:44:53 +03:00
lekss361
8be539ddc6 feat(site-finder): ПЗЗ territorial zones from Rosreestr PKK6 + zoning in analyze 2026-05-11 21:51:51 +03:00
lekss361
5b03d6d799 feat(site-finder): isochrones UI + networks VKH + OSM substations 2026-05-11 21:48:21 +03:00
lekss361
6ce634a9b9 feat(site-finder): utilities (power/pipeline) + fix Внешние факторы layout 2026-05-11 21:38:20 +03:00
lekss361
2855a01ca7 feat(site-finder): v3.5 - seasonal weather + hydrology + geotech risk 2026-05-11 21:32:07 +03:00
lekss361
7900dc5238 fix(site-finder): WKT LINESTRING needs commas + rosreestr_deals real column names 2026-05-11 20:58:49 +03:00
lekss361
1e9d32ee3c feat(site-finder): v3.2 — noise + air quality + wind analytics 2026-05-11 20:42:37 +03:00
lekss361
f419900968 feat(site-finder): v3.1 — cad_parcels_geom, analyze fallback, POI lat/lon, OSM expand
- nspd_geo: add _save_parcel() for thematic_id=1 → cad_parcels_geom (UPSERT,
  ST_Transform from Web Mercator); _persist_target now handles 1/2/5
- parcels.py: analyze endpoint geom lookup extended with cad_parcels_geom as
  3rd fallback source (after cad_quarters_geom, cad_buildings); both SELECT
  and WKT subqueries updated
- parcels.py: POI score_breakdown items now include lat/lon for map markers
- poi_loader: OSM_CATEGORIES expanded — college+university→school,
  hypermarket→shop_supermarket; coverage +3 tag pairs
2026-05-11 19:52:19 +03:00
lekss361
3aeda297ee fix(poi-sync): split Overpass query per category (504 Gateway Timeout) 2026-05-11 18:54:28 +03:00
lekss361
93f13795bb fix(poi-sync): add User-Agent header — Overpass returns 406 for python-httpx default UA 2026-05-11 18:46:47 +03:00
lekss361
6dea34186b feat(site-finder): OSM POI loader + Celery weekly sync
- New poi_loader.py: Overpass QL query for 13 POI categories in EKB bbox,
  UPSERT into osm_poi_ekb with soft 2-year filter tracking (skipped_old counter)
- New poi_sync.py Celery task wrapping sync_poi_to_db()
- celery_app.py: include poi_sync module, add poi-sync-weekly beat entry (Mon 03:00 MSK)
2026-05-11 18:08:34 +03:00
lekss361
b95559eac9 init 2026-04-25 13:45:19 +03:00