gendesign/backend/tests/test_nspd_client.py
lekss361 bc092b58f1
feat(scrapers): NSPD client foundation (#94) — search + WMS + layers (#98)
* feat(scrapers): NSPD client foundation (#94) — search_by_cad + WMS + layers

Foundation для G1 #28 ПЗЗ, G2 #29 ВРИ, G3 #30 ЗОУИТ, E1 #51 cad_parcels,
I3 #44 engineering, и поддержки on-demand #93.

Backend (`app/services/scrapers/nspd_client.py`):
- `NSPDClient` с 4 core методами:
  - `search_by_cad(cad, thematic_id)` — REST /api/geoportal/v2/search
    возвращает GeoJSON + ВРИ + land_category + cost_value за 1 запрос
  - `get_feature_info(layer_id, lon, lat, buffer_m=100)` — WMS GetFeatureInfo
    на точке: какие feature'ы layer'а покрывают (lon,lat)
  - `get_features_in_bbox(layer_id, bbox_3857)` — bulk через WMS workaround
    (WFS GetCapabilities → 404, поэтому большой bbox + центральная I/J точка)
  - `list_layers(theme_id)` — каталог слоёв в теме (PKK=1, ARN=665)
- Typed responses: NSPDFeature, NSPDSearchResult, NSPDLayer (frozen dataclasses)
- LAYERS catalog: 32 layer-id с семантическими именами (territorial_zones=875838,
  zouit_engineering=37578, risk_flooding=872205, etc) — TIER 1-6 per #94
- Coordinate helpers: lonlat_to_3857(), bbox_around_point_m()
- Reuse: HEADERS + SSL ctx + fetch_geoportal из existing nspd_lite (WAF-compatible
  urllib trick). Rate limit через rate_ms.
- WAF/Rate-limit: raises NspdLiteWafError на 403/429 — caller backoff.

Tests (`backend/tests/test_nspd_client.py`): 14 unit tests, no network.
- LAYERS catalog sanity (territorial_zones=875838 закрывает G1)
- Coordinate transforms (zero, ЕКБ center, bbox)
- NSPDFeature.from_raw parsing (full / missing fields)
- NSPDSearchResult helpers (empty/first)
- _walk_layer_tree (flat / nested / empty)
- search_by_cad через monkeypatch fetch_geoportal

Также включён micro-fix из PR #95 review (cosmetic):
- parcels.py inline comment "до 25s" → "до 15s" (matched _INLINE_FETCH_WAIT_S)

Scope NOT в этом PR (отдельные follow-up issues, sequential rule):
- Schema migrations (cad_parcels_geom +columns, nspd_territorial_zones,
  nspd_zouit, nspd_red_lines, nspd_risk_zones tables) → отдельный PR
- Celery tasks (sync_territorial_zones_bbox, sync_zouit_*, sync_risk_*) →
  отдельный PR использует client
- Admin endpoints (trigger / status) → отдельный PR
- Замена on-demand fetch на nspd_client → PR after schema ready

Closes part of #94 (foundation only — sub-issues for schema/tasks).

* fix(scrapers): address PR #98 auto-review minor feedback

Backend (nspd_client.py):
1. (line 100+) bbox_around_point_m signature split на multi-line.
2. (lazy imports) import json, time → module-level.
3. (duplicate SSL ctx) Reuse _SSL_CTX из nspd_lite через explicit import.
4. (shape defense) list_layers чекает {"data": [...]} wrapper + non-dict/list garbage → warning + empty.
5. (cad_num docstring) search_by_cad документирует формат + ссылку на validate_cad_format.

Tests: +5 mock тестов (19/19 passed):
- get_feature_info URL builder + parsing
- get_features_in_bbox bbox propagation
- list_layers_walks_tree_response
- list_layers_handles_data_wrapper
- list_layers_handles_garbage_response

Backend (parcels.py): docstring '~25с' → '~15с'.

Per auto-review on be14388.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 09:47:45 +03:00

321 lines
11 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

"""Unit tests для nspd_client (#94). Без сетевых запросов — только parsing.
Network calls (search_by_cad / get_feature_info / list_layers) проверяются
вручную через `python -c "from app.services.scrapers.nspd_client import
NSPDClient; print(NSPDClient().search_by_cad('66:41:0204016:10'))"` —
проверяется в production smoke test после deploy.
"""
from __future__ import annotations
from typing import Any
import pytest
from app.services.scrapers.nspd_client import (
LAYERS,
NSPDClient,
NSPDFeature,
NSPDLayer,
NSPDSearchResult,
_walk_layer_tree,
bbox_around_point_m,
lonlat_to_3857,
)
def test_layers_catalog_has_critical_ids() -> None:
"""Sanity check: TIER 1 critical layers зарегистрированы."""
# Из #94: TIER 1 включает 875838 (территориальные зоны / ПЗЗ G1)
assert LAYERS["territorial_zones"] == 875838
assert LAYERS["parcels"] == 36048
assert LAYERS["buildings"] == 36049
assert LAYERS["quarters"] == 36071
# TIER 2 ЗОУИТ (G3)
assert LAYERS["zouit_okn"] == 37577
assert LAYERS["zouit_engineering"] == 37578
assert LAYERS["zouit_natural"] == 37580
def test_lonlat_to_3857_zero() -> None:
"""0,0 → ~0,0 (with tiny floating error)."""
x, y = lonlat_to_3857(0.0, 0.0)
assert abs(x) < 1.0
assert abs(y) < 1.0
def test_lonlat_to_3857_ekb_center() -> None:
"""ЕКБ центр (60.605, 56.838) — known approximate projection."""
x, y = lonlat_to_3857(60.605, 56.838)
# Sanity: положительный x (восточная долгота), положительный y (северная)
assert x > 6_700_000
assert x < 6_800_000
assert y > 7_700_000
assert y < 7_800_000
def test_bbox_around_point_m_basic() -> None:
"""100m buffer вокруг точки → bbox с side ≈ 200m."""
xmin, ymin, xmax, ymax = bbox_around_point_m(60.605, 56.838, 100)
assert abs((xmax - xmin) - 200) < 1.0
assert abs((ymax - ymin) - 200) < 1.0
def test_nspd_feature_from_raw_minimal() -> None:
"""Парсинг минимального GeoJSON Feature."""
raw = {
"id": "feat-1",
"geometry": {"type": "Point", "coordinates": [1, 2]},
"properties": {"cad_num": "66:41:0204016:10"},
}
feat = NSPDFeature.from_raw(raw)
assert feat.feature_id == "feat-1"
assert feat.geometry is not None
assert feat.geometry["type"] == "Point"
assert feat.properties["cad_num"] == "66:41:0204016:10"
assert feat.crs == "EPSG:3857"
def test_nspd_feature_from_raw_missing_id() -> None:
"""Feature без id → feature_id is None."""
feat = NSPDFeature.from_raw({"geometry": None, "properties": {}})
assert feat.feature_id is None
assert feat.geometry is None
assert feat.properties == {}
def test_nspd_search_result_empty() -> None:
"""Empty result → first is None, is_empty True."""
result = NSPDSearchResult(cad_num="x", features=[], raw={})
assert result.first is None
assert result.is_empty is True
def test_nspd_search_result_nonempty() -> None:
"""First feature accessible через .first."""
feat = NSPDFeature.from_raw({"id": "x", "geometry": None, "properties": {}})
result = NSPDSearchResult(cad_num="x", features=[feat], raw={})
assert result.first is feat
assert result.is_empty is False
def test_walk_layer_tree_flat() -> None:
"""Flat list of layers — все попадают в output."""
tree: list[dict[str, Any]] = [
{"id": 1, "title": "A", "children": []},
{"id": 2, "title": "B", "children": []},
]
out = _walk_layer_tree(tree)
ids = sorted(x["id"] for x in out)
assert ids == [1, 2]
def test_walk_layer_tree_nested() -> None:
"""Nested tree — все leaves вытаскиваются."""
tree = {
"id": 1,
"title": "root",
"children": [
{"id": 2, "title": "child", "children": []},
{
"id": 3,
"title": "branch",
"children": [
{"id": 4, "title": "grandchild", "children": []},
],
},
],
}
out = _walk_layer_tree(tree)
ids = sorted(x["id"] for x in out)
# Все уровни включены — root, child, branch, grandchild
assert ids == [1, 2, 3, 4]
def test_walk_layer_tree_empty() -> None:
"""Empty input → empty output, no crash."""
assert _walk_layer_tree({}) == []
assert _walk_layer_tree([]) == []
assert _walk_layer_tree(None) == [] # type: ignore[arg-type]
def test_nspd_client_search_by_cad_parses_response(monkeypatch: pytest.MonkeyPatch) -> None:
"""search_by_cad оборачивает fetch_geoportal в NSPDSearchResult."""
fake_raw = {
"data": {
"type": "FeatureCollection",
"features": [
{
"id": "f1",
"geometry": {"type": "Polygon", "coordinates": []},
"properties": {
"cad_num": "66:41:0204016:10",
"land_record_area": 1500,
"permitted_use_established_by_document": "Многоэтажная жилая",
},
}
],
}
}
def fake_fetch(query: str, **kwargs: Any) -> dict[str, Any]:
assert query == "66:41:0204016:10"
return fake_raw
# Patch the imported reference in nspd_client (it was imported by name)
monkeypatch.setattr(
"app.services.scrapers.nspd_client.fetch_geoportal",
fake_fetch,
)
client = NSPDClient()
result = client.search_by_cad("66:41:0204016:10")
assert result.cad_num == "66:41:0204016:10"
assert len(result.features) == 1
assert result.first is not None
assert result.first.properties["permitted_use_established_by_document"] == "Многоэтажная жилая"
assert result.raw is fake_raw # preserved для debugging
def test_nspd_client_search_by_cad_empty(monkeypatch: pytest.MonkeyPatch) -> None:
"""Empty response → is_empty True, no crash."""
monkeypatch.setattr(
"app.services.scrapers.nspd_client.fetch_geoportal",
lambda *a, **kw: {"data": {"type": "FeatureCollection", "features": []}},
)
result = NSPDClient().search_by_cad("99:99:9999999")
assert result.is_empty is True
assert result.first is None
def test_nspd_layer_dataclass() -> None:
"""NSPDLayer immutable, hashable (frozen + slots)."""
layer = NSPDLayer(layer_id=875838, title="Терр.зоны", layer_type="wms", metadata={})
assert layer.layer_id == 875838
# Frozen
with pytest.raises(AttributeError):
layer.title = "x" # type: ignore[misc]
# ── Mock tests for WMS/layers methods (no network) ──────────────────────────
def test_get_feature_info_parses_response(monkeypatch: pytest.MonkeyPatch) -> None:
"""get_feature_info вызывает _http_get_json с правильным URL + парсит features."""
captured_urls: list[str] = []
def fake_http(url: str, **kwargs: Any) -> dict[str, Any]:
captured_urls.append(url)
return {
"type": "FeatureCollection",
"features": [
{
"id": "tz-1",
"geometry": {"type": "Polygon", "coordinates": []},
"properties": {"zone_code": "Ж-1"},
}
],
}
monkeypatch.setattr("app.services.scrapers.nspd_client._http_get_json", fake_http)
client = NSPDClient()
feats = client.get_feature_info(875838, lon=60.605, lat=56.838, buffer_m=100)
assert len(feats) == 1
assert feats[0].properties["zone_code"] == "Ж-1"
# Verify URL structure
assert len(captured_urls) == 1
url = captured_urls[0]
assert "/api/aeggis/v4/875838/wms?" in url
assert "REQUEST=GetFeatureInfo" in url
assert "INFO_FORMAT=application%2Fjson" in url
assert "STYLES=" in url # required per NSPD WAF
assert "CRS=EPSG%3A3857" in url
# bbox should reflect 100m buffer (200m side)
assert "BBOX=" in url
def test_get_features_in_bbox_url_builds_correctly(monkeypatch: pytest.MonkeyPatch) -> None:
"""get_features_in_bbox использует переданный bbox без преобразования."""
captured_urls: list[str] = []
def fake_http(url: str, **kwargs: Any) -> dict[str, Any]:
captured_urls.append(url)
return {"features": []}
monkeypatch.setattr("app.services.scrapers.nspd_client._http_get_json", fake_http)
client = NSPDClient()
bbox = (6700000.0, 7700000.0, 6800000.0, 7800000.0) # ЕКБ region in 3857
feats = client.get_features_in_bbox(875838, bbox)
assert feats == []
assert len(captured_urls) == 1
url = captured_urls[0]
# bbox passed directly into BBOX param (URL-encoded comma = %2C)
assert "BBOX=6700000.0%2C7700000.0%2C6800000.0%2C7800000.0" in url
# Large width/height для bulk fetch
assert "WIDTH=4096" in url
assert "HEIGHT=4096" in url
# I/J pointer в центр виртуального тайла
assert "I=2048" in url
assert "J=2048" in url
def test_list_layers_walks_tree_response(monkeypatch: pytest.MonkeyPatch) -> None:
"""list_layers вызывает /layers-theme-tree и распаковывает в NSPDLayer."""
fake_tree = [
{
"id": 1,
"title": "PKK тема",
"children": [
{
"id": 875838,
"title": "Территориальные зоны",
"layerType": "wms",
"children": [],
},
{
"id": 37577,
"title": "ЗОУИТ ОКН",
"layerType": "wms",
"children": [],
},
],
}
]
monkeypatch.setattr(
"app.services.scrapers.nspd_client._http_get_json",
lambda url, **kw: fake_tree,
)
client = NSPDClient()
layers = client.list_layers(theme_id=1)
layer_ids = [layer.layer_id for layer in layers]
# Root + 2 children = 3 leaves
assert 875838 in layer_ids
assert 37577 in layer_ids
# Sample layer metadata preserved
tz_layer = next(layer for layer in layers if layer.layer_id == 875838)
assert tz_layer.title == "Территориальные зоны"
assert tz_layer.layer_type == "wms"
def test_list_layers_handles_data_wrapper(monkeypatch: pytest.MonkeyPatch) -> None:
"""Shape defense — NSPD иногда оборачивает в {"data": [...]}."""
fake_wrapped = {
"data": [{"id": 36048, "title": "ЗУ", "children": []}],
"meta": {"count": 1},
}
monkeypatch.setattr(
"app.services.scrapers.nspd_client._http_get_json",
lambda url, **kw: fake_wrapped,
)
layers = NSPDClient().list_layers(theme_id=1)
assert len(layers) >= 1
assert any(layer.layer_id == 36048 for layer in layers)
def test_list_layers_handles_garbage_response(monkeypatch: pytest.MonkeyPatch) -> None:
"""Не dict/list response → empty list + warning log (no crash)."""
monkeypatch.setattr(
"app.services.scrapers.nspd_client._http_get_json",
lambda url, **kw: "garbage string",
)
assert NSPDClient().list_layers(theme_id=1) == []