feat(cadastre): per-category NSPD snapshot — Phase 1.5 (ZOUIT/ENK) #180

Merged

lekss361 merged 1 commit from fix/cadastre-per-category-snapshot into main

2026-05-15 13:53:49 +00:00

lekss361 commented

2026-05-15 13:49:28 +00:00

(Migrated from github.com)

Summary

Increase cadastre coverage for ZOUIT/ENK categories which were 0% in full ЕКБ job #8.

Switch base search_by_quarter from thematicSearchId=1 to thematic=1 (includes ZOUIT in features list).
Add optional category_id param routing ZOUIT (36940/469039/469040/469042) → thematicSearchId=5, ENK (39663) → thematicSearchId=15.
New Phase 1.5 in harvest_quarter: per-category probe for each PER_CATEGORY_PROBE_CATS with meta_total > 0. Skips if category ∈ GRID_WALK_CATS AND bbox is valid (grid-walk has better 225-pt coverage).

Why

Job #8 (full ЕКБ, 2408 quarters) results: parcels=353 (1.6%), buildings=10837, constructions=106, ОНС=5, ENK=0, ZOUIT=0.

Live curl probe against nspd.gov.ru:

thematicSearchId=1 returns 20-per-category but excludes ZOUIT entirely.
thematic=1 strictly better — adds ZOUIT to base response.
thematicSearchId=5 → ZOUIT-only (all 4 subtypes).
thematicSearchId=15 → ENK-only.
categoryId param is server-ignored (always returns 20 per cat mixed).
No pagination available.

Expected impact

ZOUIT coverage: 0 → up to 80 per quarter (4 subtypes × 20).
ENK coverage: 0 → up to 20 per quarter when meta non-zero.
Parcels/buildings unchanged (grid-walk still primary for overflow).
Additional NSPD calls per quarter: 0–5 (only for categories with meta_total > 0).
Backward compat: search_by_quarter(quarter) (no category_id) works exactly as before but with thematic=1 (richer response).

Test plan

pytest tests/services/test_cadastre_bulk.py tests/scrapers/test_nspd_bulk_client.py (43 passed locally)

После merge + deploy → curl-trigger pilot v7 (50 quarters):

curl -X POST https://gendsgn.ru/api/v1/admin/cadastre/jobs \
  -H "X-Admin-Token: ***" -H "Content-Type: application/json" \
  -d '{"scope":"pilot","name":"pilot_v7_per_cat"}'

Проверить SQL counts: SELECT COUNT(*) FROM cad_zouit (expect > 0), cad_enk (expect > 0).
Если pilot green → trigger ekb_full и проверить overall coverage delta.

Relates to #168.

## Summary Increase cadastre coverage for ZOUIT/ENK categories which were 0% in full ЕКБ job #8. - Switch base `search_by_quarter` from `thematicSearchId=1` to `thematic=1` (includes ZOUIT in features list). - Add optional `category_id` param routing ZOUIT (36940/469039/469040/469042) → `thematicSearchId=5`, ENK (39663) → `thematicSearchId=15`. - New **Phase 1.5** in `harvest_quarter`: per-category probe for each `PER_CATEGORY_PROBE_CATS` with `meta_total > 0`. Skips if category ∈ `GRID_WALK_CATS` AND `bbox` is valid (grid-walk has better 225-pt coverage). ## Why Job #8 (full ЕКБ, 2408 quarters) results: parcels=353 (1.6%), buildings=10837, **constructions=106**, **ОНС=5**, **ENK=0**, **ZOUIT=0**. Live `curl` probe against `nspd.gov.ru`: - `thematicSearchId=1` returns 20-per-category but **excludes ZOUIT entirely**. - `thematic=1` strictly better — adds ZOUIT to base response. - `thematicSearchId=5` → ZOUIT-only (all 4 subtypes). - `thematicSearchId=15` → ENK-only. - `categoryId` param is **server-ignored** (always returns 20 per cat mixed). - No pagination available. ## Expected impact - **ZOUIT coverage**: 0 → up to 80 per quarter (4 subtypes × 20). - **ENK coverage**: 0 → up to 20 per quarter when meta non-zero. - Parcels/buildings unchanged (grid-walk still primary for overflow). - Additional NSPD calls per quarter: 0–5 (only for categories with meta_total > 0). - Backward compat: `search_by_quarter(quarter)` (no category_id) works exactly as before but with `thematic=1` (richer response). ## Test plan - [ ] `pytest tests/services/test_cadastre_bulk.py tests/scrapers/test_nspd_bulk_client.py` (43 passed locally) - [ ] После merge + deploy → curl-trigger pilot v7 (50 quarters): ``` curl -X POST https://gendsgn.ru/api/v1/admin/cadastre/jobs \ -H "X-Admin-Token: ***" -H "Content-Type: application/json" \ -d '{"scope":"pilot","name":"pilot_v7_per_cat"}' ``` - [ ] Проверить SQL counts: `SELECT COUNT(*) FROM cad_zouit` (expect > 0), `cad_enk` (expect > 0). - [ ] Если pilot green → trigger ekb_full и проверить overall coverage delta. Relates to #168.

lekss361 commented

2026-05-15 13:53:41 +00:00

(Migrated from github.com)

Review

CI зелёный (backend 1m51s + frontend 1m42s) + cross-check (agent):

✅ Backward compat

search_by_quarter единственный production caller — bulk_harvest.py:129 (positional quarter only). Новый category_id=None default — fully backward-compat.

✅ Wire-protocol change `thematicSearchId=1` → `thematic=1`

Существующие unit-тесты (test_nspd_bulk_client.py) мокают _get_json напрямую через patch.object — param change невидим для фикстуры. Response shape {data: {features}, meta: [...]} сохраняется (mocked). Author подтвердил live-curl: оба эндпоинта возвращают identical shape, thematic=1 строго богаче (включает ЗОУИТ).

✅ Disjoint cat sets

GRID_WALK_CATS = {36368, 36369} (ЗУ/Здания) ∩ PER_CATEGORY_PROBE_CATS = {36940, 469039, 469040, 469042, 39663} (ЗОУИТ/ЕНК) = ∅. Defensive guard на L170 unreachable, но author оставил для safety — норм.

✅ Phase ordering + concurrency

Phase 1 → Phase 1.5 (serial for-loop) → Phase 2-3 (grid-walk). Global _SEMAPHORE = asyncio.Semaphore(3) ограничивает concurrent NSPD calls. Per-cat probes serial — простой и читаемый, добавляет 0–5 calls/quarter. Можно параллелить через asyncio.gather позже если нужно speed-up, но это nano-optimization.

✅ Test coverage solid

3 asyncio теста:

test_harvest_quarter_calls_per_cat_probe_for_zouit_when_meta_nonzero — probe вызывается с category_id ∈ CAT_ZOUIT, result.zouit_upserted >= 1.
test_harvest_quarter_skips_per_cat_probe_when_meta_zero — await_count == 1 (только базовый), category_id is None.
test_harvest_quarter_per_cat_probe_enk_called_when_meta_nonzero — probe с category_id=CAT_ENK.

Покрывают main happy paths + skip-condition.

✅ Drift отсутствует

QuarterSnapshot pydantic schema без изменений. TS callers отсутствуют (scraper server-side only). Routing upsert_features уже умеет различать категории.

✅ Risk profile

Default behavior строго richer (раньше ЗОУИТ был исключён, теперь включён в base snapshot). Per-cat probe — opt-in через meta_total > 0. Worst case — лишние 0–5 NSPD calls на quarter где есть ЗОУИТ/ЕНК. Pilot v7 покажет реальный delta coverage.

approve merge

## Review CI зелёный (backend 1m51s + frontend 1m42s) + cross-check (agent): ### ✅ Backward compat `search_by_quarter` единственный production caller — `bulk_harvest.py:129` (positional `quarter` only). Новый `category_id=None` default — fully backward-compat. ### ✅ Wire-protocol change `thematicSearchId=1` → `thematic=1` Существующие unit-тесты (`test_nspd_bulk_client.py`) мокают `_get_json` напрямую через `patch.object` — param change невидим для фикстуры. Response shape `{data: {features}, meta: [...]}` сохраняется (mocked). Author подтвердил live-curl: оба эндпоинта возвращают identical shape, `thematic=1` строго богаче (включает ЗОУИТ). ### ✅ Disjoint cat sets `GRID_WALK_CATS = {36368, 36369}` (ЗУ/Здания) ∩ `PER_CATEGORY_PROBE_CATS = {36940, 469039, 469040, 469042, 39663}` (ЗОУИТ/ЕНК) = ∅. Defensive guard на L170 unreachable, но author оставил для safety — норм. ### ✅ Phase ordering + concurrency Phase 1 → Phase 1.5 (serial for-loop) → Phase 2-3 (grid-walk). Global `_SEMAPHORE = asyncio.Semaphore(3)` ограничивает concurrent NSPD calls. Per-cat probes serial — простой и читаемый, добавляет 0–5 calls/quarter. Можно параллелить через `asyncio.gather` позже если нужно speed-up, но это nano-optimization. ### ✅ Test coverage solid 3 asyncio теста: - `test_harvest_quarter_calls_per_cat_probe_for_zouit_when_meta_nonzero` — probe вызывается с `category_id ∈ CAT_ZOUIT`, `result.zouit_upserted >= 1`. - `test_harvest_quarter_skips_per_cat_probe_when_meta_zero` — `await_count == 1` (только базовый), `category_id` is None. - `test_harvest_quarter_per_cat_probe_enk_called_when_meta_nonzero` — probe с `category_id=CAT_ENK`. Покрывают main happy paths + skip-condition. ### ✅ Drift отсутствует `QuarterSnapshot` pydantic schema без изменений. TS callers отсутствуют (scraper server-side only). Routing `upsert_features` уже умеет различать категории. ### ✅ Risk profile Default behavior строго **richer** (раньше ЗОУИТ был исключён, теперь включён в base snapshot). Per-cat probe — opt-in через meta_total > 0. Worst case — лишние 0–5 NSPD calls на quarter где есть ЗОУИТ/ЕНК. Pilot v7 покажет реальный delta coverage. approve merge