feat(cadastre): NSPDBulkClient + schemas (#168 PR2/5) #170

Merged

lekss361 merged 1 commit from feat/cadastre-bulk-client into main

2026-05-15 09:50:41 +00:00

lekss361 commented

2026-05-15 09:42:55 +00:00

(Migrated from github.com)

Summary

PR 2/5 of #168 Bulk cadastre ingest ЕКБ
Adds NSPDBulkClient (async, httpx) — wraps NSPD's bulk-ish endpoints with rate limiting + WAF handling
3 methods: search_by_quarter, wms_feature_info, list_objects_in_building
Pydantic schemas with extra='allow' → preserves unknown fields downstream
15 unit tests pass; 4 e2e tests (real NSPD) marked slow + skipped in CI

Files

File	Lines	Purpose
`backend/app/schemas/nspd_bulk.py`	~180	`NSPDBulkFeature`, `QuarterSnapshot`, `ObjectsListing`, `NSPDOptions`
`backend/app/scrapers/nspd_bulk_client.py`	~480	`NSPDBulkClient` async context manager + 3 methods + retry/backoff
`backend/tests/scrapers/test_nspd_bulk_client.py`	~340	15 unit + 4 e2e tests
`backend/tests/scrapers/__init__.py`	0	package marker

Endpoints wrapped (verified in HAR audit 2026-05-15)

`search_by_quarter(quarter)` → `QuarterSnapshot`

GET /api/geoportal/v2/search/geoportal?thematicSearchId=1&query={quarter}
→ data.features (20 per cat) + meta=[{categoryId, totalCount}]

Returns QuarterSnapshot.overflow_categories для категорий где totalCount > 20 (нужен grid-walk в PR3).

`wms_feature_info(layer_id, bbox_3857, click_xy, w=512, h=512)` → `list[NSPDBulkFeature]`

GET /api/aeggis/v4/{layer_id}/wms?REQUEST=GetFeatureInfo&BBOX=&I=&J=&FEATURE_COUNT=10
→ GeoJSON FeatureCollection (1+ features при попадании клика на полигон)

`list_objects_in_building(objdoc_id, registers_id=36441)` → `ObjectsListing` (Q3 deferred)

GET /api/geoportal/v1/tab-group-data?tabClass=objectsList&objdocId=X&registersId=Y
→ {flats_cad_nums: [...], parking_cad_nums: [...]}  без cap

Не используется в bulk MVP (Q3 помещений deferred). Метод готов для later.

Rate limiting + WAF

asyncio.Semaphore(3) + asyncio.sleep(0.05) jitter → ≤3 req/s sustained
429 → exponential backoff 2→4→8с (max 3 retries) → NspdBulkRateLimitError
403 → NspdBulkWafError raised immediately (no retry — нужен manual investigation)
404 / empty response → пустой QuarterSnapshot (graceful)

Headers (точно из HAR)

User-Agent: Mozilla/5.0 ... Chrome/148.0.0.0 ...
Referer: https://nspd.gov.ru/map
Accept-Language: ru-RU,ru;q=0.9,...
x-timezone: Europe/Moscow

Test plan

cd backend && uv run pytest tests/scrapers/test_nspd_bulk_client.py -v — 15 unit tests pass
e2e (requires net + permission): uv run pytest tests/scrapers/test_nspd_bulk_client.py -m slow -s -v
- test_search_by_quarter_real_ekb — 66:41:0303161 returns 117+ features, meta has 36368:170, 36369:256
- test_wms_feature_info_real — real bbox click returns 1+ features
- test_list_objects_in_building_real — building objdocId=42065602 returns >150 cad_nums
- test_rate_limit_semaphore — 5 concurrent ≤ 3 in flight

What's NOT in this PR

DB writes (PR 3 будет — bulk_harvest_quarter Celery task с upsert logic)
Grid-walker (PR 3)
Admin UI (PR 4)
Cutover into parcels/analyze flow (PR 7+ после verification)
Помещения / парковки bulk-fetch (Q3 deferred)

Закрывает 2/5 of #168.

## Summary - **PR 2/5** of [#168 Bulk cadastre ingest ЕКБ](https://github.com/lekss361/-gendesign/issues/168) - Adds `NSPDBulkClient` (async, httpx) — wraps NSPD's bulk-ish endpoints with rate limiting + WAF handling - 3 methods: `search_by_quarter`, `wms_feature_info`, `list_objects_in_building` - Pydantic schemas with `extra='allow'` → preserves unknown fields downstream - 15 unit tests pass; 4 e2e tests (real NSPD) marked slow + skipped in CI ## Files | File | Lines | Purpose | |---|---|---| | `backend/app/schemas/nspd_bulk.py` | ~180 | `NSPDBulkFeature`, `QuarterSnapshot`, `ObjectsListing`, `NSPDOptions` | | `backend/app/scrapers/nspd_bulk_client.py` | ~480 | `NSPDBulkClient` async context manager + 3 methods + retry/backoff | | `backend/tests/scrapers/test_nspd_bulk_client.py` | ~340 | 15 unit + 4 e2e tests | | `backend/tests/scrapers/__init__.py` | 0 | package marker | ## Endpoints wrapped (verified in HAR audit 2026-05-15) ### `search_by_quarter(quarter)` → `QuarterSnapshot` ``` GET /api/geoportal/v2/search/geoportal?thematicSearchId=1&query={quarter} → data.features (20 per cat) + meta=[{categoryId, totalCount}] ``` Returns `QuarterSnapshot.overflow_categories` для категорий где `totalCount > 20` (нужен grid-walk в PR3). ### `wms_feature_info(layer_id, bbox_3857, click_xy, w=512, h=512)` → `list[NSPDBulkFeature]` ``` GET /api/aeggis/v4/{layer_id}/wms?REQUEST=GetFeatureInfo&BBOX=&I=&J=&FEATURE_COUNT=10 → GeoJSON FeatureCollection (1+ features при попадании клика на полигон) ``` ### `list_objects_in_building(objdoc_id, registers_id=36441)` → `ObjectsListing` (Q3 deferred) ``` GET /api/geoportal/v1/tab-group-data?tabClass=objectsList&objdocId=X&registersId=Y → {flats_cad_nums: [...], parking_cad_nums: [...]} без cap ``` **Не используется в bulk MVP** (Q3 помещений deferred). Метод готов для later. ## Rate limiting + WAF - `asyncio.Semaphore(3)` + `asyncio.sleep(0.05)` jitter → ≤3 req/s sustained - 429 → exponential backoff 2→4→8с (max 3 retries) → `NspdBulkRateLimitError` - 403 → `NspdBulkWafError` raised immediately (no retry — нужен manual investigation) - 404 / empty response → пустой `QuarterSnapshot` (graceful) ## Headers (точно из HAR) ```python User-Agent: Mozilla/5.0 ... Chrome/148.0.0.0 ... Referer: https://nspd.gov.ru/map Accept-Language: ru-RU,ru;q=0.9,... x-timezone: Europe/Moscow ``` ## Test plan - [x] `cd backend && uv run pytest tests/scrapers/test_nspd_bulk_client.py -v` — 15 unit tests pass - [ ] e2e (requires net + permission): `uv run pytest tests/scrapers/test_nspd_bulk_client.py -m slow -s -v` - `test_search_by_quarter_real_ekb` — `66:41:0303161` returns 117+ features, meta has 36368:170, 36369:256 - `test_wms_feature_info_real` — real bbox click returns 1+ features - `test_list_objects_in_building_real` — building objdocId=42065602 returns >150 cad_nums - `test_rate_limit_semaphore` — 5 concurrent ≤ 3 in flight ## What's NOT in this PR - DB writes (PR 3 будет — `bulk_harvest_quarter` Celery task с upsert logic) - Grid-walker (PR 3) - Admin UI (PR 4) - Cutover into `parcels/analyze` flow (PR 7+ после verification) - Помещения / парковки bulk-fetch (Q3 deferred) Закрывает 2/5 of #168.

lekss361 commented

2026-05-15 09:50:30 +00:00

(Migrated from github.com)

Final review (SHA `70c079d`, CI ✅)

Cross-checked агентом — self-contained PR без DB/frontend coupling, безопасно мержить ahead of #169.

Verified by agent

✅ Targets main directly — не stacked на #169. Diff = только 4 Python файла (schemas + client + tests + init). Никакого SQL, никаких runtime dependencies на блокированный #169.

✅ Pydantic schemas align с #169 columns: cad_num, quarter_cad_number, area, cost_value, cost_index, purpose, floors, year_built, readable_address, ownership_type, right_type, status, objdoc_id, registers_id + extra='allow' для raw_props JSONB. Нет PR #147-style field drift.

✅ Async httpx client design:

asyncio.Semaphore(3) + 50ms jitter rate-limit
429 → exponential backoff 2/4/8s → NspdBulkRateLimitError
403 → NspdBulkWafError immediately (no retry, manual investigation)
404 → empty QuarterSnapshot (graceful)
httpx.Timeout(30, connect=10)
HAR-audited headers (User-Agent, Referer, x-timezone)

✅ No DB writes / SAVEPOINT / :param::cast — все DB write logic deferred в PR3. Этот PR — pure client library.

✅ Tests — 15 unit (mocked httpx.get): schema parse, overflow_categories, 404→empty, 403→WAF, 429→backoff exhaustion, semaphore max-concurrent ≤3, WMS parse. 4 e2e marked slow+skipped в CI. Coverage decent.

✅ Backend + frontend CI green.

Минор (не блокер, follow-up note)

Module-level _SEMAPHORE shared across instances в одном process, но multiple Celery worker processes → effective concurrency = 3 × workers. Acceptable, но note для PR3 когда integration в celery task.
Тест touches private _SEMAPHORE._value — fragile но isolated.
flats_count redundant с len(flats_cad_nums).

Note about sequencing

PR3 (integration с bulk_harvest_quarter Celery task) потребует исправленного #169 schema. То есть #169 blockers (DROP cad_parcels_geom + K.3 rename) должны быть зафикшены до PR3 merge. Но #170 — pure library, safe to merge independently.

approve merge

## Final review (SHA `70c079d`, CI ✅) Cross-checked агентом — self-contained PR без DB/frontend coupling, безопасно мержить ahead of #169. ### Verified by agent ✅ **Targets main directly** — не stacked на #169. Diff = только 4 Python файла (schemas + client + tests + init). Никакого SQL, никаких runtime dependencies на блокированный #169. ✅ **Pydantic schemas align** с #169 columns: `cad_num`, `quarter_cad_number`, `area`, `cost_value`, `cost_index`, `purpose`, `floors`, `year_built`, `readable_address`, `ownership_type`, `right_type`, `status`, `objdoc_id`, `registers_id` + `extra='allow'` для `raw_props JSONB`. **Нет PR #147-style field drift**. ✅ **Async httpx client design**: - `asyncio.Semaphore(3)` + 50ms jitter rate-limit - 429 → exponential backoff 2/4/8s → `NspdBulkRateLimitError` - 403 → `NspdBulkWafError` immediately (no retry, manual investigation) - 404 → empty QuarterSnapshot (graceful) - `httpx.Timeout(30, connect=10)` - HAR-audited headers (User-Agent, Referer, x-timezone) ✅ **No DB writes / SAVEPOINT / `:param::cast`** — все DB write logic deferred в PR3. Этот PR — pure client library. ✅ **Tests** — 15 unit (mocked httpx.get): schema parse, overflow_categories, 404→empty, 403→WAF, 429→backoff exhaustion, semaphore max-concurrent ≤3, WMS parse. 4 e2e marked slow+skipped в CI. Coverage decent. ✅ **Backend + frontend CI green**. ### Минор (не блокер, follow-up note) - Module-level `_SEMAPHORE` shared across instances в одном process, но multiple Celery worker **processes** → effective concurrency = 3 × workers. Acceptable, но note для PR3 когда integration в celery task. - Тест touches private `_SEMAPHORE._value` — fragile но isolated. - `flats_count` redundant с `len(flats_cad_nums)`. ### Note about sequencing PR3 (integration с `bulk_harvest_quarter` Celery task) потребует **исправленного** #169 schema. То есть #169 blockers (DROP cad_parcels_geom + K.3 rename) должны быть зафикшены до PR3 merge. Но #170 — pure library, safe to merge independently. approve merge