lekss361
81cf274e80
fix( #234 ): address re-review nits — Settings singleton + docstring sync
...
1. _acquire_harvest_lock: Settings() instance → singleton 'settings' import
from app.core.config — соответствует convention (scrape_kn.py, objective.py,
celery_app.py все используют singleton).
2. _trigger_harvest docstring: убрана incorrect фраза «остальные ... видят
harvest_eta_seconds от первого запроса» (код возвращает eta=None для
denied-lock callers). Заменена на actual behavior: graceful no-data UI
с ПКК fallback на FE.
PR #250 re-review followup (issues #3 + docstring sync).
2026-05-17 09:31:43 +03:00
lekss361
b99ac76463
Merge remote-tracking branch 'forgejo/main' into fix/234-nspd-harvest-eta-badge
...
# Conflicts:
# backend/app/services/site_finder/quarter_dump_lookup.py
2026-05-17 09:19:12 +03:00
lekss361
990c109756
fix( #234 ): NSPD harvest ETA badge + SETNX dedupe (Sub-PR C)
...
Backend (quarter_dump_lookup.py):
- _acquire_harvest_lock: Redis SETNX TTL=120s на quarter, защищает от burst
N concurrent analyze, ставящих N одинаковых harvest task в очередь
- _trigger_harvest: использует lock перед apply_async, возвращает False если
lock уже взят (другой запрос триггернул раньше)
- make_empty_result/EMPTY_DUMP_RESULT: новое поле harvest_eta_seconds в
nspd_dump dict, типичный harvest_quarter = 60с
- /analyze: пробрасывает поле через nspd_dump dict (нет typed schema —
response_model=None для /analyze endpoint, dict уходит как есть)
Frontend (NspdFreshnessBadge, NspdZoningBlock):
- Countdown «НСПД: загрузка ~Nс» вместо бесконечного спиннера
- После остановки countdown (remaining=0) NspdZoningBlock показывает
«загрузка дольше обычного» + ссылку на ПКК вместо infinite skeleton
Tests: 5 новых unit + 2 для empty_result schema (всего +7, pass)
Closes #234 (UX-side; data-side resolves когда Sub-PR B + D merged).
2026-05-17 09:13:06 +03:00
10b79da2e9
fix( #243 ): cad_zouit fallback срабатывает когда nspd dump полностью отсутствует ( #248 )
Deploy / changes (push) Successful in 6s
Deploy / deploy (push) Successful in 46s
Deploy / build-backend (push) Successful in 1m36s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m44s
2026-05-17 06:03:19 +00:00
lekss361
f70be684da
feat(#29,#232): wire cad_parcels.permitted_use in analyze + cad_zouit fallback (G2+G3)
...
TASK A (#29 G2): add parcel_meta to analyze response
- New ParcelMeta Pydantic schema in app/schemas/parcel.py
- SELECT from cad_parcels WHERE cad_num=:c in analyze_parcel() (step 9f)
- Returns permitted_use_established_by_document, land_record_category_type,
land_record_subtype, cost_value; None when row absent
- Tests: test_analyze_parcel_meta.py (found + not-found cases)
TASK B (#232 G3): cad_zouit fallback in _get_zouit_overlaps
- When nspd_quarter_dumps has zouit_count==0, fall back to ST_Intersects
query on cad_zouit (3483 rows, GIST indexed)
- Overlaps tagged with source='cad_zouit'; format compatible with NSPD path
- gate_verdict.py: BLOCKER_TYPE_ZONE_KEYWORDS tuple for keyword-based
classification (охранная зона / трубопровод / электр / газ -> blocker;
СЗЗ -> warning); NSPD subcategory path preserved backward-compat
- Tests: 6 new test cases in test_gate_verdict.py covering cad_zouit path
and backward-compat for NSPD subcategory path
Updated db.execute call sequence in test_analyze_*.py (index shift +1 at pos 10).
2026-05-17 08:17:22 +03:00
lekss361
fdb54834e5
fix(nspd): rename migration 89→98, fix red lines ST_Area→ST_Length ( #220 )
...
- data/sql/98_*: rename from 89 to avoid collision with existing 89_drop_dead_brin
- _get_red_lines: ST_Length(ST_Intersection(planar)::geography) instead of
ST_Area(ST_Intersection(::geography, ::geography)) — fixes PostGIS 3.4
tolerance error and returns correct non-zero length for LINESTRING intersections
- Rename intersection_area_sqm → intersection_length_m across schema, TS types,
frontend component, and tests; add test_get_red_lines_db_exception_returns_empty
Addresses review-bot feedback on PR #220 .
2026-05-16 19:33:14 +03:00
lekss361
ed3c128528
feat(nspd): TIER 4 opportunity layers + red lines ( #94 PR2 of 4)
...
- NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt)
+ QuarterDump.opportunity field
- nspd_sync: harvest_quarter accepts include_opportunity, denorm cols
has_auction_parcels + opportunity_count in UPSERT
- quarter_dump_lookup:
- _get_opportunity_parcels (sort by distance, early-exit on count=0)
- _get_red_lines (query existing dump.red_lines core path, layer='red_lines')
- SQL 89_*: has_auction_parcels + opportunity_count + partial index
- Pydantic: OpportunityParcel + RedLine schemas
- /analyze: nspd_opportunity_parcels + nspd_red_lines fields
- Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration
- Tests: 28 total (11 PR1 + 17 PR2), all pass
Red lines uses existing core harvest path (layer 879243 already in dump.red_lines
from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed.
Part of #94
2026-05-16 19:06:22 +03:00
lekss361
1595157ebf
feat(nspd): enable TIER 3 risk zones harvest + extract in analyze ( #94 PR1)
...
- harvest_stale_quarters fanout passes include_risks=True to all queued tasks
- _get_risk_zones() with ST_Intersects + ST_Area intersection calc
- _extract_features_by_layer() generic helper for future PR2+ reuse
- RiskZone Pydantic schema; /analyze returns nspd_risk_zones list
- NspdRiskZonesBlock component with severity color-coding (red/yellow/orange)
- 11 unit tests for extract/intersect/null-area/db-exception paths
Part of #94 (TIER 3 risk layers — flooding, landslide, burns, erosion etc)
2026-05-16 16:11:59 +03:00
4deb1b0284
feat(parcels): connection-points endpoint (Forgejo #115 Phase 1 backend) ( #190 )
Deploy / changes (push) Successful in 5s
Deploy / build-frontend (push) Failing after 0s
Deploy / build-worker (push) Successful in 5m11s
Deploy / deploy (push) Failing after 0s
Deploy / build-backend (push) Successful in 3m44s
2026-05-16 05:58:27 +00:00
lekss361
1c2f80a4b8
feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel ( #94 Sprint 1.1 FINAL) ( #116 )
...
* feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 #4 FINAL)
Замыкает Sprint 1.1 из #94 part 2 plan. После этого PR пользователь видит
свежие НСПД данные в UI (frontend integration — отдельный PR).
Backend (new app/services/site_finder/quarter_dump_lookup.py):
- `derive_quarter_cad(cad_num)` — 3/4/5-сегмент → quarter (3-segment)
- `get_quarter_dump_data(db, cad_num, parcel_wkt)` — main entrypoint:
- Reads nspd_quarter_dumps row для derived quarter
- Freshness threshold: 180 days
- Missing/stale/harvest_error → trigger harvest_quarter.apply_async() fire-
and-forget (lazy import против circular), return EMPTY_DUMP_RESULT
- Fresh + parcel_wkt=None → metadata only (no spatial queries)
- Fresh + geometry → 3 spatial queries via jsonb_array_elements + ST_Transform
(3857→4326) + ST_Intersects / ST_DWithin
- 3 private helpers:
- `_get_zoning` — point-in-polygon parcel centroid vs territorial_zones, LIMIT 1
- `_get_zouit_overlaps` — все zouit_% layers пересекающиеся с parcel
- `_get_engineering_nearby` — engineering_structures в 200m, sorted by distance
- `EMPTY_DUMP_RESULT` module-level constant — DRY для no-dump fallback (used
in get_quarter_dump_data internal + analyze_parcel try/except wrap)
Backend (parcels.py):
- Import EMPTY_DUMP_RESULT + get_quarter_dump_data
- Call wrapped в try/except — если nspd_quarter_dumps недоступна (DB timeout
/ table missing) → EMPTY_DUMP_RESULT fallback вместо 500 (consistent с
resilience pattern других optional fetches)
- Response gets 4 new fields:
- nspd_zoning: dict | None (G1 ПЗЗ — zone_code, zone_name, source)
- nspd_zouit_overlaps: list[dict] (G3 — overlaps в parcel, per ЗОУИТ group)
- nspd_engineering_nearby: list[dict] (I3 — engineering structures в 200m)
- nspd_dump: dict (freshness metadata — available, fetched_at_utc, stale,
harvest_triggered, total_features)
Tests: 13 new в test_quarter_dump_lookup.py (mock-based, no real DB):
- derive_quarter_cad 5 edge cases (3seg, 4seg, 5seg, invalid, whitespace)
- get_quarter_dump no_row → harvest triggered
- stale (>180d) → harvest triggered, stale=True
- harvest_error row → retry harvest triggered
- parcel_wkt=None → metadata only (1 DB call)
- fresh + zoning extraction
- fresh + zouit_overlaps list
- fresh integration: все 4 keys present
47 pre-existing tests still pass.
Code review (code-reviewer pre-push): MINOR, 0 blocking. Applied 2 of 4:
- ✅ #1 : try/except wrap around get_quarter_dump_data в analyze_parcel
(защита от DB unavailability) + DRY через EMPTY_DUMP_RESULT module const
- ✅ #2 : removed redundant nspd_zoning.fetched_at_utc (DRY — freshness в
nspd_dump.fetched_at_utc)
- ⏭ Deferred (acceptable): #3 ad-hoc harvest_quarter retry cooldown для
harvest_error rows (только при high traffic + persistent NSPD errors);
#4 raw_props в response — tech debt, убрать вместе с frontend PR
Performance note: 3 spatial queries per analyze adds ~10-50ms on typical
~100-feature quarter. Mitigation if quarters grow dense: materialized
per-layer sub-table (отдельная DB issue).
Closes Sprint 1.1 part of #94 . Frontend rendering этих 4 полей — отдельный
PR (next: #112 / #115 / #114 ).
* fix(site-finder): address PR #116 auto-review M1-M5
M1 (mutation risk): replace EMPTY_DUMP_RESULT direct refs with
_make_empty_result() factory. dict(...) shallow copy left nested
nspd_dump shared by reference across concurrent requests — single
mutation pollutes module sentinel for all subsequent calls. Now
каждый caller gets independent dict.
M2 (O(N) spatial scan): SELECT extended denormalized counts
(territorial_zones_count, zouit_count, engineering_count). Each
spatial helper accepts layer_counts and early-returns when count=0
— skips heavy jsonb_array_elements + ST_Transform/ST_Intersects
scan entirely. Critical для quarters с 2000+ features.
M4 (documentation): _trigger_harvest docstring describes known
burst/no-dedup limitation + TODO Redis SETNX (отдельный PR).
M5 (test fragility): _make_db_mock_with_spatial docstring describes
positional-call contract — db.execute order (0=dump, 1=zoning, 2=zouit,
3=engineering) и зависимость от count-values.
+4 new tests (17 total, all pass):
- test_make_empty_result_returns_independent_copies (mutation safety)
- test_make_empty_result_overrides
- test_early_exit_all_counts_zero_no_spatial_queries
- test_early_exit_partial_counts
Per auto-review on 3068a9c#116 review
M1 residual fix: parcels.py exception path использовал EMPTY_DUMP_RESULT
singleton ref вместо factory. Сейчас readonly access, но нарушает
documented invariant модуля.
Rename `_make_empty_result` → `make_empty_result` (public API), import в
parcels.py, использовать в try/except fallback. Каждый request получает
независимый dict — никаких shared references.
M4 (Redis SETNX dedup) + M5 (test fragility) — deferred per review,
documented в code/issue. Acceptable trade-offs:
- M4: UPSERT idempotency делает данные safe; burst-duplicate task'и тратят
WAF traffic впустую но не повреждают данные.
- M5: docstring contractually describes positional-call order.
17/17 tests pass. ruff/format clean.
Per auto-review on aef8308
2026-05-13 09:14:19 +03:00
