lekss361
|
f3b0ce5687
|
feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4 part A) (#111)
* feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4/4 part A)
Sprint 1.1 item #3 из #94 part 2. Замыкает harvest cycle: Celery task
оборачивает NSPDClient.search_by_quarter (PR #109) + UPSERT в
nspd_quarter_dumps (PR #110). После — analyze_parcel будет читать dumps
вместо on-demand NSPD HTTP в request-цикле (Sprint 1.1 #4, следующий PR).
Backend:
- New `backend/app/workers/tasks/nspd_sync.py`:
- `harvest_quarter(quarter_cad, region_code=66, include_zouit=True,
include_risks=False)` — single-quarter harvest. autoretry_for=
(NspdLiteWafError,) с retry_backoff=True, max_retries=3,
soft_time_limit=120s. Generic exception → upsert error row с
harvest_error=str(exc), counts=0 (НЕ re-raise).
- `harvest_stale_quarters(region_code, max_age_days=90, batch_size=50)` —
beat fanout: SELECT cad_quarters_geom EXCEPT fresh dumps → apply_async per cad.
- UPSERT использует ST_Multi(ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON,3857),4326))
для quarter_geom (schema MultiPolygon strict per database-expert contract).
- `celery_app.py`:
- Add `nspd_sync` в `include`
- Beat entry `nspd-harvest-stale-quarters` Mon 04:00 МСК, batch_size=50,
max_age_days=90
- `worker_ready` sanity check `SELECT 1 FROM nspd_quarter_dumps LIMIT 0`
(non-fatal critical log per Bug_Worker_Ready_EarlyReturn_Fixed pattern)
- `admin_scrape.py`:
- `HarvestQuarterRequest` Pydantic с pattern `^\d+:\d+:\d+$` (quarter only)
- `POST /api/v1/admin/scrape/nspd/harvest-quarter` manual trigger,
requires X-Admin-Token, returns task_id.
Tests (`test_nspd_sync.py`): 14 mock-based (no real NSPD/DB):
- happy path: search_by_quarter mock → verify SQL params/structure
- empty quarter → row with quarter_geom=NULL, total=0, no harvest_error
- WAF error → autoretry kicks in (raise re-propagated)
- Generic error → upsert error row
- Fanout: 50 stale → 50 apply_async calls
- Layer names preservation (all 5 zouit + 11 risks if enabled)
- Geometry SRID handling
Code review (code-reviewer pre-push): APPROVE, 0 blocking, 5 minor deferred:
- inline datetime import in error branch (style)
- layers_fetched array literal via string concat (current layer names safe)
- harvest_stale_quarters error rows always re-tried (intended, undocumented)
- test call_args.kwargs fragile (works for current callsite)
- beat collision with refresh-ekb-districts-medians every ~2 years (negligible)
Part of #94. Next: Sprint 1.1 #4 — analyze_parcel reads dump cache.
* fix(nspd-sync): address PR #111 auto-review M1-M5+M7 findings
M1 (runtime bug): remove WHERE region_code from outer cad_quarters_geom SELECT
in harvest_stale_quarters — column does not exist in prod schema
(cad_number, geom, raw_props, fetched_at, source). Would crash on first beat
tick. Inner nspd_quarter_dumps subquery keeps region_code filter unchanged.
M2 (correctness): _upsert_dump теперь делает db.rollback() в except перед
db.close() — prevents PendingRollbackError на следующем session use, если
PostGIS отбракует geometry или JSON cast failure.
M3 (low): worker_ready breadcrumb INSERT тоже теперь имеет rollback —
consistency с остальными session usages в celery_app.py.
M4 (style): layers_fetched — заменил manual string concat
'{' + ','.join(...) + '}' на native Python list. psycopg v3 сам сериализует
list → text[]. Защита от future layer names с запятыми/фигурными скобками.
M5 (style): import datetime as _dt поднят из else branch в top-level imports.
M7 (test cleanup): удалил dead __enter__/__exit__ mock setup в
test_harvest_stale_quarters_fanout / _empty. harvest_stale_quarters не
использует session как context manager — moc'и no-op.
14/14 tests pass. ruff/format clean.
Per auto-review on 9b2a289.
---------
Co-authored-by: lekss361 <claudestars@proton.me>
|
2026-05-12 18:45:57 +03:00 |
|
