lekss361
ed3c128528
feat(nspd): TIER 4 opportunity layers + red lines ( #94 PR2 of 4)
...
- NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt)
+ QuarterDump.opportunity field
- nspd_sync: harvest_quarter accepts include_opportunity, denorm cols
has_auction_parcels + opportunity_count in UPSERT
- quarter_dump_lookup:
- _get_opportunity_parcels (sort by distance, early-exit on count=0)
- _get_red_lines (query existing dump.red_lines core path, layer='red_lines')
- SQL 89_*: has_auction_parcels + opportunity_count + partial index
- Pydantic: OpportunityParcel + RedLine schemas
- /analyze: nspd_opportunity_parcels + nspd_red_lines fields
- Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration
- Tests: 28 total (11 PR1 + 17 PR2), all pass
Red lines uses existing core harvest path (layer 879243 already in dump.red_lines
from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed.
Part of #94
2026-05-16 19:06:22 +03:00
lekss361
1595157ebf
feat(nspd): enable TIER 3 risk zones harvest + extract in analyze ( #94 PR1)
...
- harvest_stale_quarters fanout passes include_risks=True to all queued tasks
- _get_risk_zones() with ST_Intersects + ST_Area intersection calc
- _extract_features_by_layer() generic helper for future PR2+ reuse
- RiskZone Pydantic schema; /analyze returns nspd_risk_zones list
- NspdRiskZonesBlock component with severity color-coding (red/yellow/orange)
- 11 unit tests for extract/intersect/null-area/db-exception paths
Part of #94 (TIER 3 risk layers — flooding, landslide, burns, erosion etc)
2026-05-16 16:11:59 +03:00
lekss361
d104222643
fix(cadastre,nspd_sync): SQL syntax bugs surfaced by ekb_full v2 ( #186 )
...
Three distinct SQL syntax errors caught from worker logs:
1. **upsert_zouit** (bulk_harvest.py:940) — trailing comma after 4326
in ST_Transform(..., 3857), 4326,) → 'syntax error at or near ")"'.
Hit on 8+ quarters in 0108 block (66:41:0108055/61/69/77 etc).
Likely introduced by ruff-format auto-comma on closing paren.
2. **nspd_sync** (nspd_sync.py:120) — `:fetched_at_utc::timestamptz`
psycopg double-colon trap → 'syntax error at or near ":"'.
Crashed legacy on-demand harvest_quarter called from analyze.
3. **job_settings** (job_settings.py:224) — `:extra_config::jsonb`
same double-colon trap in admin settings UPDATE.
All converted to CAST(:x AS type) pattern (psycopg v3 safe).
40 tests still passing.
Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 19:18:43 +03:00
lekss361
f3b0ce5687
feat(nspd): harvest_quarter Celery task + beat + admin endpoint ( #94 pt.4 part A) ( #111 )
...
* feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4/4 part A)
Sprint 1.1 item #3 из #94 part 2. Замыкает harvest cycle: Celery task
оборачивает NSPDClient.search_by_quarter (PR #109 ) + UPSERT в
nspd_quarter_dumps (PR #110 ). После — analyze_parcel будет читать dumps
вместо on-demand NSPD HTTP в request-цикле (Sprint 1.1 #4 , следующий PR).
Backend:
- New `backend/app/workers/tasks/nspd_sync.py`:
- `harvest_quarter(quarter_cad, region_code=66, include_zouit=True,
include_risks=False)` — single-quarter harvest. autoretry_for=
(NspdLiteWafError,) с retry_backoff=True, max_retries=3,
soft_time_limit=120s. Generic exception → upsert error row с
harvest_error=str(exc), counts=0 (НЕ re-raise).
- `harvest_stale_quarters(region_code, max_age_days=90, batch_size=50)` —
beat fanout: SELECT cad_quarters_geom EXCEPT fresh dumps → apply_async per cad.
- UPSERT использует ST_Multi(ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON,3857),4326))
для quarter_geom (schema MultiPolygon strict per database-expert contract).
- `celery_app.py`:
- Add `nspd_sync` в `include`
- Beat entry `nspd-harvest-stale-quarters` Mon 04:00 МСК, batch_size=50,
max_age_days=90
- `worker_ready` sanity check `SELECT 1 FROM nspd_quarter_dumps LIMIT 0`
(non-fatal critical log per Bug_Worker_Ready_EarlyReturn_Fixed pattern)
- `admin_scrape.py`:
- `HarvestQuarterRequest` Pydantic с pattern `^\d+:\d+:\d+$` (quarter only)
- `POST /api/v1/admin/scrape/nspd/harvest-quarter` manual trigger,
requires X-Admin-Token, returns task_id.
Tests (`test_nspd_sync.py`): 14 mock-based (no real NSPD/DB):
- happy path: search_by_quarter mock → verify SQL params/structure
- empty quarter → row with quarter_geom=NULL, total=0, no harvest_error
- WAF error → autoretry kicks in (raise re-propagated)
- Generic error → upsert error row
- Fanout: 50 stale → 50 apply_async calls
- Layer names preservation (all 5 zouit + 11 risks if enabled)
- Geometry SRID handling
Code review (code-reviewer pre-push): APPROVE, 0 blocking, 5 minor deferred:
- inline datetime import in error branch (style)
- layers_fetched array literal via string concat (current layer names safe)
- harvest_stale_quarters error rows always re-tried (intended, undocumented)
- test call_args.kwargs fragile (works for current callsite)
- beat collision with refresh-ekb-districts-medians every ~2 years (negligible)
Part of #94 . Next: Sprint 1.1 #4 — analyze_parcel reads dump cache.
* fix(nspd-sync): address PR #111 auto-review M1-M5+M7 findings
M1 (runtime bug): remove WHERE region_code from outer cad_quarters_geom SELECT
in harvest_stale_quarters — column does not exist in prod schema
(cad_number, geom, raw_props, fetched_at, source). Would crash on first beat
tick. Inner nspd_quarter_dumps subquery keeps region_code filter unchanged.
M2 (correctness): _upsert_dump теперь делает db.rollback() в except перед
db.close() — prevents PendingRollbackError на следующем session use, если
PostGIS отбракует geometry или JSON cast failure.
M3 (low): worker_ready breadcrumb INSERT тоже теперь имеет rollback —
consistency с остальными session usages в celery_app.py.
M4 (style): layers_fetched — заменил manual string concat
'{' + ','.join(...) + '}' на native Python list. psycopg v3 сам сериализует
list → text[]. Защита от future layer names с запятыми/фигурными скобками.
M5 (style): import datetime as _dt поднят из else branch в top-level imports.
M7 (test cleanup): удалил dead __enter__/__exit__ mock setup в
test_harvest_stale_quarters_fanout / _empty. harvest_stale_quarters не
использует session как context manager — moc'и no-op.
14/14 tests pass. ruff/format clean.
Per auto-review on 9b2a289
2026-05-12 18:45:57 +03:00
