lekss361
|
3beb75d68a
|
fix(cadastre): skip grid-walk for broken quarter geometries (#168) (#179)
* fix(cadastre): skip grid-walk for broken quarter geometries (#168)
Pilot v5 (50 quarters) разоблачил что cad_quarters_geom имеет 1735/2408 (72%)
ЕКБ rows с broken micro-precision geom (width ~0.01м instead of 200-4000м).
Causes:
- grid-walk WMS receives bbox ~1mm × 1mm → 500 ServiceException на all 450 req
- One quarter hung 2+ min (silent failure in async loop)
Real fix: re-import cad_quarters_geom geometry from NSPD (category 36381).
Tracked в follow-up issue.
Quick fix (this PR): sanity-check quarter_bbox_3857 — return None если width
ИЛИ height not in [100m, 10000m]. Bulk_harvest_quarter then skips grid-walk
для broken quarters, только snapshot phase запускается (20 per cat).
After this fix:
- All 50 pilot quarters получают snapshot (parcels/buildings/etc up to 20 each)
- Grid-walk runs only для 673/2408 healthy quarters → no 500 cascades, no hangs
* test(cadastre): add quarter_bbox_3857 boundary tests (PR #179 bot blocker)
- healthy bbox 750m -> returns tuple
- width <100m (broken micro-precision) -> None
- width >10000m (oversized) -> None
- height <100m (symmetry) -> None
- row missing (quarter not in cad_quarters_geom) -> None
---------
Co-authored-by: lekss361 <claudestars@proton.me>
|
2026-05-15 16:25:06 +03:00 |
|
lekss361
|
5da66bbc93
|
feat(cadastre): bulk_harvest worker + grid-walker + admin API (#168 PR3/5) (#171)
* feat(cadastre): bulk_harvest_quarter Celery task + grid-walker + saga state (#168 PR3/5)
Add bulk cadastre harvest pipeline:
- services/cadastre/bulk_harvest.py: async harvest_quarter() orchestrator (4 phases)
+ 7 upsert helpers (parcels/buildings/constructions/oncs/enks/zouit/quarter_stats)
using CAST(:x AS jsonb) pattern, begin_nested() SAVEPOINT per grid-walk upsert
- services/cadastre/grid_geometry.py: quarter_bbox_3857 (PostGIS ST_Extent)
+ generate_grid_click_points (15x15 = 225 sub-bbox grid)
- workers/tasks/scrape_cadastre.py: bulk_harvest_quarter_task (acks_late=True,
dont_autoretry_for NspdBulkWafError), enqueue_cadastre_harvest, cleanup_zombies
- api/v1/admin_cadastre.py: 5 endpoints behind AdminTokenAuth — create/list/get/cancel/resume
- Tests: 13 service unit + 8 API tests
* fixup(cadastre): drop importorskip (PR2 merged) — imports now top-level (#168 PR3)
* fixup(cadastre): tile_size unified param + register slow marker (#168 PR3)
Blocker #1: rename tile_width/tile_height → tile_size in generate_grid_click_points.
Callers in bulk_harvest.py and test_cadastre_bulk.py used tile_size; def had
tile_width+tile_height → TypeError at runtime.
Blocker #2 (10 failures in test_admin_cadastre.py) was side-effect of #1:
TypeError at import chain (app.main → admin_cadastre → bulk_harvest)
broke FastAPI app load → dependency_overrides AttributeError. Now resolved.
Also register `slow` pytest marker in pyproject.toml to suppress
PytestUnknownMarkWarning.
* fixup(cadastre): use importlib.util.find_spec instead of import (#168 PR3)
Root cause Blocker #2: `import app.workers.tasks.scrape_cadastre` at module
top-level REBOUND `app` from FastAPI instance (line 14) to the Python package.
All subsequent `app.dependency_overrides[get_db] = ...` failed with
AttributeError because `app` was now the namespace module.
Fix: probe module availability with importlib.util.find_spec — does not
import or rebind names. FastAPI `app` instance stays intact.
All 28 cadastre tests pass locally.
---------
Co-authored-by: lekss361 <claudestars@proton.me>
|
2026-05-15 13:31:32 +03:00 |
|