Commit graph

4 commits

Author SHA1 Message Date
lekss361
df5034fe29 feat(cadastre): per-category NSPD snapshot — Phase 1.5 (ZOUIT/ENK)
Switch base search_by_quarter from thematicSearchId=1 to thematic=1
(includes ZOUIT in features). Add optional category_id param that routes
ZOUIT (36940/469039/469040/469042) -> thematicSearchId=5,
ENK (39663) -> thematicSearchId=15.

Add Phase 1.5 in harvest_quarter: for each PER_CATEGORY_PROBE_CATS
category present in meta_counts (meta_total>0), run dedicated
per-category search call. Skips if category in GRID_WALK_CATS and
bbox is valid (grid-walk gives better coverage).

Curl-probe confirmed NSPD API caps at 20/category with no pagination;
categoryId param is server-ignored; thematic=1 strictly better than
thematicSearchId=1 (includes ZOUIT).

Closes part of #168 (ZOUIT/ENK coverage gap).
2026-05-15 16:48:59 +03:00
lekss361
3beb75d68a
fix(cadastre): skip grid-walk for broken quarter geometries (#168) (#179)
* fix(cadastre): skip grid-walk for broken quarter geometries (#168)

Pilot v5 (50 quarters) разоблачил что cad_quarters_geom имеет 1735/2408 (72%)
ЕКБ rows с broken micro-precision geom (width ~0.01м instead of 200-4000м).
Causes:
- grid-walk WMS receives bbox ~1mm × 1mm → 500 ServiceException на all 450 req
- One quarter hung 2+ min (silent failure in async loop)

Real fix: re-import cad_quarters_geom geometry from NSPD (category 36381).
Tracked в follow-up issue.

Quick fix (this PR): sanity-check quarter_bbox_3857 — return None если width
ИЛИ height not in [100m, 10000m]. Bulk_harvest_quarter then skips grid-walk
для broken quarters, только snapshot phase запускается (20 per cat).

After this fix:
- All 50 pilot quarters получают snapshot (parcels/buildings/etc up to 20 each)
- Grid-walk runs only для 673/2408 healthy quarters → no 500 cascades, no hangs

* test(cadastre): add quarter_bbox_3857 boundary tests (PR #179 bot blocker)

- healthy bbox 750m -> returns tuple
- width <100m (broken micro-precision) -> None
- width >10000m (oversized) -> None
- height <100m (symmetry) -> None
- row missing (quarter not in cad_quarters_geom) -> None

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 16:25:06 +03:00
lekss361
99e2210919
fix(cadastre): NSPD response parse — properties.category + top-level meta (#168) (#177)
* fix(cadastre): NSPD response parse — properties.category + top-level meta (#168)

* fix(cadastre): test fixtures use real NSPD shape — properties.category + top-level meta (#168)

Bot review of #177 flagged that fixtures still used legacy properties.categoryId
+ data.meta — meaning CI exercised only fallback branches, not primary paths
where the actual bug lived. Pattern of 3 consecutive "live-only" parse fixes
(#175 verify, #176 headers, #177 parse) confirmed need for test coverage.

Changes:
- test_nspd_bulk_client.py: sample_quarter_response → properties.category +
  meta moved to top-level (real NSPD shape verified live)
- test_cadastre_bulk.py: 7 fixtures categoryId → category (regex replace)
- test_nspd_bulk_feature_parse_basic: primary path now exercised

Plus schema hardening per bot review:
- NSPDBulkFeature.category_id: `is not None` check (not truthy `or`)
  to avoid edge case category=0; int() wrapped in try/except so
  non-numeric ID (e.g. "ЗУ") doesn't crash upsert_features loop.

35/35 tests pass locally.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 15:40:01 +03:00
lekss361
5da66bbc93
feat(cadastre): bulk_harvest worker + grid-walker + admin API (#168 PR3/5) (#171)
* feat(cadastre): bulk_harvest_quarter Celery task + grid-walker + saga state (#168 PR3/5)

Add bulk cadastre harvest pipeline:
- services/cadastre/bulk_harvest.py: async harvest_quarter() orchestrator (4 phases)
  + 7 upsert helpers (parcels/buildings/constructions/oncs/enks/zouit/quarter_stats)
  using CAST(:x AS jsonb) pattern, begin_nested() SAVEPOINT per grid-walk upsert
- services/cadastre/grid_geometry.py: quarter_bbox_3857 (PostGIS ST_Extent)
  + generate_grid_click_points (15x15 = 225 sub-bbox grid)
- workers/tasks/scrape_cadastre.py: bulk_harvest_quarter_task (acks_late=True,
  dont_autoretry_for NspdBulkWafError), enqueue_cadastre_harvest, cleanup_zombies
- api/v1/admin_cadastre.py: 5 endpoints behind AdminTokenAuth — create/list/get/cancel/resume
- Tests: 13 service unit + 8 API tests

* fixup(cadastre): drop importorskip (PR2 merged) — imports now top-level (#168 PR3)

* fixup(cadastre): tile_size unified param + register slow marker (#168 PR3)

Blocker #1: rename tile_width/tile_height → tile_size in generate_grid_click_points.
Callers in bulk_harvest.py and test_cadastre_bulk.py used tile_size; def had
tile_width+tile_height → TypeError at runtime.

Blocker #2 (10 failures in test_admin_cadastre.py) was side-effect of #1:
TypeError at import chain (app.main → admin_cadastre → bulk_harvest)
broke FastAPI app load → dependency_overrides AttributeError. Now resolved.

Also register `slow` pytest marker in pyproject.toml to suppress
PytestUnknownMarkWarning.

* fixup(cadastre): use importlib.util.find_spec instead of import (#168 PR3)

Root cause Blocker #2: `import app.workers.tasks.scrape_cadastre` at module
top-level REBOUND `app` from FastAPI instance (line 14) to the Python package.
All subsequent `app.dependency_overrides[get_db] = ...` failed with
AttributeError because `app` was now the namespace module.

Fix: probe module availability with importlib.util.find_spec — does not
import or rebind names. FastAPI `app` instance stays intact.

All 28 cadastre tests pass locally.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 13:31:32 +03:00