lekss361
fa724f60ac
fix(cadastre): count snapshot+per-cat requests in cadastre_jobs.requests_count
...
scrape_cadastre.py previously incremented requests_count only by
grid_walk_requests, so snapshot-only quarters (and the entire ekb_full
job after PR #179 skips grid-walk on broken geom) reported 0 NSPD
requests — misleading for observability.
Add HarvestResult.snapshot_requests counter (Phase 1 search + Phase 1.5
per-cat probes) and a total_requests property. Update Celery task to
write total_requests into cadastre_jobs.requests_count.
Tests added for the new property + Phase 1 snapshot_requests=1 assertion.
2026-05-15 17:10:03 +03:00
lekss361
d5b57ccbf1
feat(cadastre): per-category NSPD snapshot — Phase 1.5 (ZOUIT/ENK) ( #180 )
...
Switch base search_by_quarter from thematicSearchId=1 to thematic=1
(includes ZOUIT in features). Add optional category_id param that routes
ZOUIT (36940/469039/469040/469042) -> thematicSearchId=5,
ENK (39663) -> thematicSearchId=15.
Add Phase 1.5 in harvest_quarter: for each PER_CATEGORY_PROBE_CATS
category present in meta_counts (meta_total>0), run dedicated
per-category search call. Skips if category in GRID_WALK_CATS and
bbox is valid (grid-walk gives better coverage).
Curl-probe confirmed NSPD API caps at 20/category with no pagination;
categoryId param is server-ignored; thematic=1 strictly better than
thematicSearchId=1 (includes ZOUIT).
Closes part of #168 (ZOUIT/ENK coverage gap).
Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 16:53:49 +03:00
lekss361
3beb75d68a
fix(cadastre): skip grid-walk for broken quarter geometries ( #168 ) ( #179 )
...
* fix(cadastre): skip grid-walk for broken quarter geometries (#168 )
Pilot v5 (50 quarters) разоблачил что cad_quarters_geom имеет 1735/2408 (72%)
ЕКБ rows с broken micro-precision geom (width ~0.01м instead of 200-4000м).
Causes:
- grid-walk WMS receives bbox ~1mm × 1mm → 500 ServiceException на all 450 req
- One quarter hung 2+ min (silent failure in async loop)
Real fix: re-import cad_quarters_geom geometry from NSPD (category 36381).
Tracked в follow-up issue.
Quick fix (this PR): sanity-check quarter_bbox_3857 — return None если width
ИЛИ height not in [100m, 10000m]. Bulk_harvest_quarter then skips grid-walk
для broken quarters, только snapshot phase запускается (20 per cat).
After this fix:
- All 50 pilot quarters получают snapshot (parcels/buildings/etc up to 20 each)
- Grid-walk runs only для 673/2408 healthy quarters → no 500 cascades, no hangs
* test(cadastre): add quarter_bbox_3857 boundary tests (PR #179 bot blocker)
- healthy bbox 750m -> returns tuple
- width <100m (broken micro-precision) -> None
- width >10000m (oversized) -> None
- height <100m (symmetry) -> None
- row missing (quarter not in cad_quarters_geom) -> None
---------
Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 16:25:06 +03:00
lekss361
6dfff5e046
fix(cadastre): CAST :geom AS text — psycopg AmbiguousParameter ( #168 ) ( #178 )
...
Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 16:02:03 +03:00
lekss361
5da66bbc93
feat(cadastre): bulk_harvest worker + grid-walker + admin API ( #168 PR3/5) ( #171 )
...
* feat(cadastre): bulk_harvest_quarter Celery task + grid-walker + saga state (#168 PR3/5)
Add bulk cadastre harvest pipeline:
- services/cadastre/bulk_harvest.py: async harvest_quarter() orchestrator (4 phases)
+ 7 upsert helpers (parcels/buildings/constructions/oncs/enks/zouit/quarter_stats)
using CAST(:x AS jsonb) pattern, begin_nested() SAVEPOINT per grid-walk upsert
- services/cadastre/grid_geometry.py: quarter_bbox_3857 (PostGIS ST_Extent)
+ generate_grid_click_points (15x15 = 225 sub-bbox grid)
- workers/tasks/scrape_cadastre.py: bulk_harvest_quarter_task (acks_late=True,
dont_autoretry_for NspdBulkWafError), enqueue_cadastre_harvest, cleanup_zombies
- api/v1/admin_cadastre.py: 5 endpoints behind AdminTokenAuth — create/list/get/cancel/resume
- Tests: 13 service unit + 8 API tests
* fixup(cadastre): drop importorskip (PR2 merged) — imports now top-level (#168 PR3)
* fixup(cadastre): tile_size unified param + register slow marker (#168 PR3)
Blocker #1 : rename tile_width/tile_height → tile_size in generate_grid_click_points.
Callers in bulk_harvest.py and test_cadastre_bulk.py used tile_size; def had
tile_width+tile_height → TypeError at runtime.
Blocker #2 (10 failures in test_admin_cadastre.py) was side-effect of #1 :
TypeError at import chain (app.main → admin_cadastre → bulk_harvest)
broke FastAPI app load → dependency_overrides AttributeError. Now resolved.
Also register `slow` pytest marker in pyproject.toml to suppress
PytestUnknownMarkWarning.
* fixup(cadastre): use importlib.util.find_spec instead of import (#168 PR3)
Root cause Blocker #2 : `import app.workers.tasks.scrape_cadastre` at module
top-level REBOUND `app` from FastAPI instance (line 14) to the Python package.
All subsequent `app.dependency_overrides[get_db] = ...` failed with
AttributeError because `app` was now the namespace module.
Fix: probe module availability with importlib.util.find_spec — does not
import or rebind names. FastAPI `app` instance stays intact.
All 28 cadastre tests pass locally.
---------
Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 13:31:32 +03:00
