lekss361/gendesign
1
0
Fork 0
Code Issues 57 Pull requests Projects Releases Packages Wiki Activity Actions
228 commits 226 branches 0 tags 42 MiB
Commit graph

27 commits

Author SHA1 Message Date
lekss361
0e2d976f9a feat(parcels): layout signature schemas + extractor (#113 PR A) 
Phase 2.1 minimal foundation for #113 Layout Analysis (Макс feedback 12.05).
Без layout_type/balcony_count (отсутствуют в БД, ждут B2B Объектив #52).

Added:
- 7 Pydantic schemas в schemas/parcel.py: LayoutSignature, BestLayoutsRequest,
  TopLayoutRow, LayoutTzMixRow, LayoutTzRecommendation, LayoutDataQuality,
  BestLayoutsResponse
- services/site_finder/layout_signature.py: room_bucket_from_flat /
  area_bin / layout_signature (3 pure-Python helpers, no DB)
- tests/test_layout_signature.py: 27 unit tests (parametrized)

No DB / API changes — foundation для PR B (mv_layout_velocity) и PR C
(POST /best-layouts endpoint).
 2026-05-16 10:49:10 +03:00
lekss361
4deb1b0284 feat(parcels): connection-points endpoint (Forgejo #115 Phase 1 backend) (#190)
Some checks failed
Deploy / changes (push) Successful in 5s
Details
Deploy / build-frontend (push) Failing after 0s
Details
Deploy / build-worker (push) Successful in 5m11s
Details
Deploy / deploy (push) Failing after 0s
Details
Deploy / build-backend (push) Successful in 3m44s
Details
2026-05-16 05:58:27 +00:00
lekss361
e561df1b55 feat(parcels): competitors endpoint (Forgejo #112 Phase 1 backend) (#191)
Some checks failed
CI / backend (push) Successful in 1m16s
Details
Deploy / changes (push) Successful in 6s
Details
Deploy / build-backend (push) Failing after 12s
Details
Deploy / build-worker (push) Failing after 12s
Details
Deploy / build-frontend (push) Failing after 0s
Details
Deploy / deploy (push) Failing after 0s
Details
CI / frontend (push) Successful in 2m17s
Details
2026-05-15 22:21:05 +00:00
lekss361
50547dcbb3 test(deploy): validate Forgejo Actions end-to-end flow (#189)
Some checks failed
CI / backend (push) Successful in 1m40s
Details
Deploy / changes (push) Successful in 5s
Details
Deploy / build-backend (push) Failing after 11s
Details
Deploy / build-worker (push) Failing after 11s
Details
CI / frontend (push) Successful in 2m13s
Details
Deploy / build-frontend (push) Failing after 12s
Details
Deploy / deploy (push) Failing after 0s
Details
2026-05-15 21:55:04 +00:00
lekss361
4d492ebc50
perf(cadastre): freshness skip + grid-walk heartbeat + log noise reduction (#187) 
* perf(cadastre): freshness skip + grid-walk heartbeat + reduce WMS log noise

Three operational improvements after ekb_full v3 completed at 99.1%:

Fix H — per-quarter freshness skip (scrape_cadastre.py):
- enqueue_cadastre_harvest queries cad_quarter_stats.fetched_at per quarter
- Skip quarters where fetched_at >= NOW() - skip_fresh_hours (default 1h)
- extra_config.skip_fresh_hours=0 disables skip (full re-fetch)
- Saves ~30k NSPD calls on repeated runs of same scope

Fix I — heartbeat in grid-walk loop (bulk_harvest.py:_grid_walk_category):
- 225 cells x 70-300ms = 15-75s heartbeat silence
- Pass update_progress callback to _grid_walk_category
- Call every heartbeat_every (50) cells with phase progress info
- Prevents cleanup_zombies false-positive cancellation

Fix J — NSPD WMS HTTP 500 noise to debug (bulk_harvest.py:_grid_walk_category):
- NSPD WMS endpoint frequently returns HTTP 500 (server-side issue)
- Was: logger.warning each, 200+ warnings per quarter
- Now: logger.debug (visible via --log-level=DEBUG)

40 tests passing.

* fix(cadastre): parse NSPD DD.MM.YYYY dates (Fix K — job 18 root cause)

Job 18 (manual_list recovery 11 quarters) crashed every quarter with:
  psycopg.errors.DatetimeFieldOverflow: date/time field value out of range: '13.03.2023'

NSPD returns registration_date in mixed formats:
- DD.MM.YYYY for Сооружения / ОНС / ЕНК / ЗОУИТ
- YYYY-MM-DD ISO for ЗУ / Здания
- YYYY-MM-DD HH:MM:SS with time

PG CAST(:date AS date) accepts ISO only — DD.MM.YYYY crashes.

Add _parse_nspd_date helper: handles both formats + None/empty.
Apply in upsert_construction, upsert_onc, upsert_enk, upsert_zouit, upsert_parcel
for registration_date / build_record_registration_date / legal_act_date.

3 new unit tests for the helper (43 tests total).

* fix(cadastre): address bot blockers — pivot Fix H to raw_targets + tests

Bot review on PR #187 caught:
1. cadastre_jobs has no extra_config column → Fix H crashed on KeyError
2. Missing tests for Fix H + Fix I
3. _parse_nspd_date didn't validate date ranges

Changes:
- Pivot Fix H: read skip_fresh_hours from raw_targets JSONB (existing column),
  no schema migration needed
- _parse_nspd_date: validate via date.fromisoformat() — rejects 2023-13-45
- Add 3 tests for Fix H in backend/tests/workers/test_scrape_cadastre.py
- Add 2 tests for Fix I heartbeat callbacks in test_cadastre_bulk.py
- Add 1 test for parse_nspd_date range validation (was: only format)

49 tests passing.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 23:02:37 +03:00
lekss361
5f8df655cc
fix(cadastre): cad_parcels.geom Polygon -> MultiPolygon (migration 93) (#185) 
NSPD returns MultiPolygon for Многоконтурный участок (e.g. 66:41:0105017:4)
which crashed upsert_parcel — schema was strict POLYGON.

Migration 93: ALTER COLUMN geom Polygon -> MultiPolygon USING ST_Multi(geom),
DROP+RECREATE GIST index + back-compat VIEW cad_parcels_geom.

upsert_parcel SQL wraps ST_Transform in ST_Multi() — coerces Polygon to
MultiPolygon. NSPD returns both types (simple + multi-contour parcels).

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 18:52:38 +03:00
lekss361
e3359c4f03
fix(cadastre): filter Point/LineString geometry in upsert_parcel/zouit (#184) 
cad_parcels.geom is geometry(Polygon, 4326) — strict schema. NSPD
occasionally returns Point geometry for parcels without detailed
boundary, causing INSERT failure:
  psycopg.errors.InvalidParameterValue:
  Geometry type (Point) does not match column type (Polygon)

This killed pilot v8 at quarter 66:41:0104002 — worker autoretry
exhausted, job hung at 25/50 (heartbeat stale).

Fix: filter geometry.type at Python level in upsert_parcel + upsert_zouit
(same pattern as upsert_quarter_geom_from_feature). Non-Polygon
geometry → geom=NULL, raw_props preserved.

Tables with permissive GEOMETRY schema (cad_buildings, cad_constructions,
cad_enk, cad_oncs) unaffected — they accept any geometry type.

Tests: 3 new (Point parcel → geom=NULL, Polygon happy path, LineString zouit).

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 18:37:29 +03:00
lekss361
e695bca1e4
fix(cadastre): remove shared phase_state early-exit (race condition) (#183) 
cadastre_jobs.phase_state is shared between ALL parallel Celery workers
of the same job. The early-exit triggered as soon as the FIRST worker
wrote phase=done via progress_cb — all subsequent workers (reading same
shared row) bailed without work.

Effect: pilot v7 made 5 NSPD requests for 50 quarters (=0.1/quarter).
Full ekb_full job #8: 4950 req / 2408 quarters = 2/quarter — 95% workers
early-exited without doing snapshot phase.

Root cause of 1.6% parcels coverage. Fix A/B/C only partially helped
because most workers never reached Phase 1/1.5.

Solution: delete the early-exit. Idempotency is guaranteed by
ON CONFLICT DO UPDATE in every upsert_* — re-enqueued tasks
write same rows without duplicates.

Closes part of #168 (root-cause fix for race condition revealed by #182 metrics).

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 17:41:27 +03:00
lekss361
057f8891dc
fix(cadastre): count snapshot+per-cat requests in cadastre_jobs.requests_count (#182) 
scrape_cadastre.py previously incremented requests_count only by
grid_walk_requests, so snapshot-only quarters (and the entire ekb_full
job after PR #179 skips grid-walk on broken geom) reported 0 NSPD
requests — misleading for observability.

Add HarvestResult.snapshot_requests counter (Phase 1 search + Phase 1.5
per-cat probes) and a total_requests property. Update Celery task to
write total_requests into cadastre_jobs.requests_count.

Tests added for the new property + Phase 1 snapshot_requests=1 assertion.

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 17:17:01 +03:00
lekss361
905ef443fc
fix(cadastre): auto-heal cad_quarters_geom from CAT_QUARTER_STATS feature (#181) 
72% of EKB quarters (1735/2408) have broken micro-precision geom in
cad_quarters_geom. NSPD returns valid quarter polygon in CAT_QUARTER_STATS
(36381) feature but Phase 4 was only consuming stats.

Add upsert_quarter_geom_from_feature: CTE-based UPDATE that transforms
3857 polygon to 4326 and writes only when existing geom is NULL or broken
(bbox width outside 100-10000m), verified via SQL-side sanity check.
Wire into Phase 4 with begin_nested savepoint: malformed NSPD GeoJSON
(self-intersecting ring) rollbacks only geom UPDATE, preserves
upsert_quarter_stats.

Closes part of #168 (follow-up data quality fix for #179).

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 17:16:39 +03:00
lekss361
d5b57ccbf1
feat(cadastre): per-category NSPD snapshot — Phase 1.5 (ZOUIT/ENK) (#180) 
Switch base search_by_quarter from thematicSearchId=1 to thematic=1
(includes ZOUIT in features). Add optional category_id param that routes
ZOUIT (36940/469039/469040/469042) -> thematicSearchId=5,
ENK (39663) -> thematicSearchId=15.

Add Phase 1.5 in harvest_quarter: for each PER_CATEGORY_PROBE_CATS
category present in meta_counts (meta_total>0), run dedicated
per-category search call. Skips if category in GRID_WALK_CATS and
bbox is valid (grid-walk gives better coverage).

Curl-probe confirmed NSPD API caps at 20/category with no pagination;
categoryId param is server-ignored; thematic=1 strictly better than
thematicSearchId=1 (includes ZOUIT).

Closes part of #168 (ZOUIT/ENK coverage gap).

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 16:53:49 +03:00
lekss361
3beb75d68a
fix(cadastre): skip grid-walk for broken quarter geometries (#168) (#179) 
* fix(cadastre): skip grid-walk for broken quarter geometries (#168)

Pilot v5 (50 quarters) разоблачил что cad_quarters_geom имеет 1735/2408 (72%)
ЕКБ rows с broken micro-precision geom (width ~0.01м instead of 200-4000м).
Causes:
- grid-walk WMS receives bbox ~1mm × 1mm → 500 ServiceException на all 450 req
- One quarter hung 2+ min (silent failure in async loop)

Real fix: re-import cad_quarters_geom geometry from NSPD (category 36381).
Tracked в follow-up issue.

Quick fix (this PR): sanity-check quarter_bbox_3857 — return None если width
ИЛИ height not in [100m, 10000m]. Bulk_harvest_quarter then skips grid-walk
для broken quarters, только snapshot phase запускается (20 per cat).

After this fix:
- All 50 pilot quarters получают snapshot (parcels/buildings/etc up to 20 each)
- Grid-walk runs only для 673/2408 healthy quarters → no 500 cascades, no hangs

* test(cadastre): add quarter_bbox_3857 boundary tests (PR #179 bot blocker)

- healthy bbox 750m -> returns tuple
- width <100m (broken micro-precision) -> None
- width >10000m (oversized) -> None
- height <100m (symmetry) -> None
- row missing (quarter not in cad_quarters_geom) -> None

---------

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 16:25:06 +03:00
lekss361
99e2210919
fix(cadastre): NSPD response parse — properties.category + top-level meta (#168) (#177) 
* fix(cadastre): NSPD response parse — properties.category + top-level meta (#168)

* fix(cadastre): test fixtures use real NSPD shape — properties.category + top-level meta (#168)

Bot review of #177 flagged that fixtures still used legacy properties.categoryId
+ data.meta — meaning CI exercised only fallback branches, not primary paths
where the actual bug lived. Pattern of 3 consecutive "live-only" parse fixes
(#175 verify, #176 headers, #177 parse) confirmed need for test coverage.

Changes:
- test_nspd_bulk_client.py: sample_quarter_response → properties.category +
  meta moved to top-level (real NSPD shape verified live)
- test_cadastre_bulk.py: 7 fixtures categoryId → category (regex replace)
- test_nspd_bulk_feature_parse_basic: primary path now exercised

Plus schema hardening per bot review:
- NSPDBulkFeature.category_id: `is not None` check (not truthy `or`)
  to avoid edge case category=0; int() wrapped in try/except so
  non-numeric ID (e.g. "ЗУ") doesn't crash upsert_features loop.

35/35 tests pass locally.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 15:40:01 +03:00
lekss361
608490fbba
fix(parcels): _parse_floors handle int (post-migration #169) (#176) 
* fix(parcels): _parse_floors handle int (post-migration #169 schema change)

After PR #169 cad_buildings schema migration, `floors` column is INT
(was TEXT in legacy schema). Existing `_parse_floors(r.get("floors"))`
call in analyze_parcel → _neighbors_summary crashes with:
  AttributeError: 'int' object has no attribute 'strip'

Fix: type union str | int | None. If int → return directly (no strip).
Preserve TEXT range parsing ("5-7" → 7) for backwards-compat with
any legacy data still in cad_buildings_old_apr26.

* test(smoke): production smoke tests for post-deploy regressions (#168)

Add tests/smoke/test_prod_smoke.py covering known regression surfaces.
Marks: prod_smoke + slow. Env: PROD_SMOKE_BASE_URL, PROD_SMOKE_ADMIN_TOKEN.
Run manually: cd backend && uv run pytest tests/smoke/ -m prod_smoke -v

* fix(cadastre): exact-match headers to nspd_lite to bypass NSPD WAF (#168)

Pilot v2 (job_id=2) failed 50/50 with HTTP 403 WAF block. Comparing
nspd_bulk_client.DEFAULT_HEADERS vs legacy nspd_lite.HEADERS (which works
on VPS IP since April 2026):

  PascalCase keys → lowercase keys
  Chrome/148 UA → Chrome/144 UA
  No cache-control / pragma → "no-cache" both
  accept-language ru first → en first
  No origin → "https://nspd.gov.ru"
  referer "/map" → "/map?thematic=PKK"

NSPD WAF (BotShield-class) likely fingerprints на header order + values
combined with TLS fingerprint. Matching legacy exactly minimizes deltas.

Test plan: retry pilot job after deploy, expect 0 WAF blocks for first
5 quarters.

* fix(test): exclude prod_smoke tests by default (#168)

CI ran tests/smoke/test_prod_smoke.py and they hit production
https://gendsgn.ru/api/v1/parcels/.../analyze which currently returns 500
(parse_floors regression — exactly what this PR fixes). Catch-22: PR can't
merge because smoke tests fail against pre-merge prod.

Fix: add `addopts = ["-m", "not prod_smoke"]` so default pytest excludes
them. Run manually post-deploy with: pytest -m prod_smoke -v

---------

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 15:08:16 +03:00
lekss361
5da66bbc93
feat(cadastre): bulk_harvest worker + grid-walker + admin API (#168 PR3/5) (#171) 
* feat(cadastre): bulk_harvest_quarter Celery task + grid-walker + saga state (#168 PR3/5)

Add bulk cadastre harvest pipeline:
- services/cadastre/bulk_harvest.py: async harvest_quarter() orchestrator (4 phases)
  + 7 upsert helpers (parcels/buildings/constructions/oncs/enks/zouit/quarter_stats)
  using CAST(:x AS jsonb) pattern, begin_nested() SAVEPOINT per grid-walk upsert
- services/cadastre/grid_geometry.py: quarter_bbox_3857 (PostGIS ST_Extent)
  + generate_grid_click_points (15x15 = 225 sub-bbox grid)
- workers/tasks/scrape_cadastre.py: bulk_harvest_quarter_task (acks_late=True,
  dont_autoretry_for NspdBulkWafError), enqueue_cadastre_harvest, cleanup_zombies
- api/v1/admin_cadastre.py: 5 endpoints behind AdminTokenAuth — create/list/get/cancel/resume
- Tests: 13 service unit + 8 API tests

* fixup(cadastre): drop importorskip (PR2 merged) — imports now top-level (#168 PR3)

* fixup(cadastre): tile_size unified param + register slow marker (#168 PR3)

Blocker #1: rename tile_width/tile_height → tile_size in generate_grid_click_points.
Callers in bulk_harvest.py and test_cadastre_bulk.py used tile_size; def had
tile_width+tile_height → TypeError at runtime.

Blocker #2 (10 failures in test_admin_cadastre.py) was side-effect of #1:
TypeError at import chain (app.main → admin_cadastre → bulk_harvest)
broke FastAPI app load → dependency_overrides AttributeError. Now resolved.

Also register `slow` pytest marker in pyproject.toml to suppress
PytestUnknownMarkWarning.

* fixup(cadastre): use importlib.util.find_spec instead of import (#168 PR3)

Root cause Blocker #2: `import app.workers.tasks.scrape_cadastre` at module
top-level REBOUND `app` from FastAPI instance (line 14) to the Python package.
All subsequent `app.dependency_overrides[get_db] = ...` failed with
AttributeError because `app` was now the namespace module.

Fix: probe module availability with importlib.util.find_spec — does not
import or rebind names. FastAPI `app` instance stays intact.

All 28 cadastre tests pass locally.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 13:31:32 +03:00
lekss361
b7889d113b
feat(cadastre): NSPDBulkClient + schemas for bulk ingest (#168 PR2/5) (#170) 
Add async NSPD client (nspd_bulk_client.py) with 3 new methods:
- search_by_quarter: REST /api/geoportal/v2/search/geoportal — snapshot 20 per cat + meta totals
- wms_feature_info: WMS GetFeatureInfo для grid-walk discovery
- list_objects_in_building: tab-group-data (Q3 deferred — код ready)

Rate limit: asyncio.Semaphore(3) + 0.05s jitter.
429 → exponential backoff 2→4→8s, max 3 retries.
403 (WAF) → raise immediately, no retry.

Pydantic schemas (nspd_bulk.py): NSPDBulkFeature, QuarterSnapshot, ObjectsListing
with extra='allow' to preserve unknown fields → raw_props JSONB downstream.

15 unit tests + 4 e2e (real NSPD, skipped in CI by default).

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 12:50:41 +03:00
lekss361
d7fbaa0528
feat(velocity): per-room-bucket breakdown in VelocityResult (#163) 
* fix(velocity): :window_interval::interval cast syntax — same bug as PR #152

SQLAlchemy text() parser confuses ':window_interval' (named param) followed
by '::interval' (cast operator) — exactly как было в PR #152 с :weights::jsonb.

Result: psycopg видит literal ':window_interval' в SQL → syntax error →
exception caught by velocity catch → return None → UI shows null.

## Fix

':window_interval::interval' → 'CAST(:window_interval AS interval)' (2 мест:
sales_rows query + _get_ekb_median percentile).

Pre-push code-reviewer должен был catch — добавим в feedback что для каждого
text() SQL grep ':[a-z]*::' before push.

Refs: PR #158 deploy verify, e2e velocity = null root cause

* feat(velocity): add per-room-bucket breakdown to VelocityResult

Add third SQL query (bucket_rows) aggregating deals_total_count by
room_bucket ('студия','1','2','3','4+') across mapped competitors.

New fields:
- VelocityResult.by_room_bucket: aggregate {units, sqm, complexes_count}
- sample_competitors[].by_room_bucket: {bucket: units}

Bucket query wrapped в SAVEPOINT — failure degrades gracefully к empty
dict без abort outer tx.

CAST(:window_interval AS interval) pattern per PR #160 lesson.

Test coverage: 3 new tests (aggregation, empty fallback, sample entries).

Closes part of #161

---------

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 10:09:03 +03:00
lekss361
8fa1d005fd
feat(velocity): migrate D2 sales source → objective_corpus_room_month (#158) 
* feat(velocity): migrate D2 sales source from stale sale_graph to objective_corpus_room_month

Closes #156

* fix(velocity): cross-stack contract + NULL safety per pre-push review

3 pre-push code-reviewer findings fixed:

1. TS union extended ('objective'|'sale_graph'|'rosreestr_fallback')
   + UI conditions handle both objective и sale_graph как valid sources.
2. COALESCE(deals_total_vol_m2, deals_total_count * 45.0) — NULL safety
   for DKP-only rows (vol_m2 nullable, count > 0).
3. room_bucket parking filter — verified false positive (все 5 buckets
   apartments: студия/1/2/3/4+).

Refs: PR #157 pre-push code-reviewer

* fix(types): add 'objective' to velocity_source union (cherry-pick miss)

Cherry-pick от закрытого PR #157 потерял этот файл. Frontend type-check fails
с 'no overlap' error. Restoring.

Refs: PR #158 frontend CI failure

---------

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 09:34:00 +03:00
lekss361
e93bb6121e
feat(site-finder): D2 velocity-score from domrf_kn_sale_graph (#34 sub-PR 1/2) (#146) 
compute_velocity service queries competitor sales в радиусе 3км:
- ST_MakePoint(longitude, latitude) — domrf_kn_objects не имеет geom column
- JOIN domrf_kn_sale_graph за 6 мес (area_sq primary, realised*45 fallback)
- Normalize vs ЕКБ-wide median → velocity_score 0..1
- confidence: high/medium/low (competitors_count + months_observed)
- Top 5 sample competitors для UI

Integration: analyze_parcel.response['velocity'] top-level field.

Schema corrections vs spec:
- obj_name → comm_name
- region_code → region_cd
- contracted (INT) → area_sq (м²)

Tests: 102/102 pass.
Vault: Module_Velocity_Service.md NEW.

Closes #144 (sub-PR 2 frontend закроет #34)

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 01:24:53 +03:00
lekss361
76a88b2827
feat(site-finder): gate verdict aggregator — can-build-MKD signal (#32 G5 sub-PR 1/2) (#142) 
Pure-function aggregator collapses nspd_zoning / nspd_zouit_overlaps /
nspd_engineering_nearby / nspd_dump в один GateVerdict TypedDict.

Logic:
- ПЗЗ не Ж-* → BLOCKER
- ЗОУИТ sub=17 (инжен. охранная) → BLOCKER
- Другие ЗОУИТ → WARNING
- Нет инжен. сетей в 200м → WARNING
- nspd_dump stale → source: nspd_dump_partial
- Нет dump → can_build_mkd: 'unknown'

Integration: новое поле gate_verdict в analyze_parcel response.

Tests: 15/15 pass (mock-based).

Vault: code/modules/Module_Gate_Verdict.md NEW.

Closes #140 (sub-PR 2 frontend закроет #32)

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 00:59:53 +03:00
lekss361
7250aa6187
feat(site-finder): weight profiles endpoints + analyze integration (#114 sub-PR 3/4) (#138) 
Admin CRUD router + non-breaking analyze_parcel integration. 46 tests pass.

Refs: #114

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 00:40:06 +03:00
lekss361
5aea78c2d8
feat(site-finder): weight profiles service — Pydantic + CRUD (#114 sub-PR 2/4) (#137) 
Per #114 sub-PR 2/4. Pydantic v2 models + CRUD service. 14 passing tests.

API: list/get/get_default/create/update/delete_profile + resolve_weights
fallback to system defaults.

Validation: ALLOWED_CATEGORIES guard + weight bounds [-2.0, 3.0].
Default-uniqueness в одной transaction (job_settings.py pattern).

Vault: code/modules/Module_Weight_Profiles_Service.md.

Refs: #114

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 00:29:37 +03:00
lekss361
1c2f80a4b8
feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 FINAL) (#116) 
* feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 #4 FINAL)

Замыкает Sprint 1.1 из #94 part 2 plan. После этого PR пользователь видит
свежие НСПД данные в UI (frontend integration — отдельный PR).

Backend (new app/services/site_finder/quarter_dump_lookup.py):
- `derive_quarter_cad(cad_num)` — 3/4/5-сегмент → quarter (3-segment)
- `get_quarter_dump_data(db, cad_num, parcel_wkt)` — main entrypoint:
  - Reads nspd_quarter_dumps row для derived quarter
  - Freshness threshold: 180 days
  - Missing/stale/harvest_error → trigger harvest_quarter.apply_async() fire-
    and-forget (lazy import против circular), return EMPTY_DUMP_RESULT
  - Fresh + parcel_wkt=None → metadata only (no spatial queries)
  - Fresh + geometry → 3 spatial queries via jsonb_array_elements + ST_Transform
    (3857→4326) + ST_Intersects / ST_DWithin
- 3 private helpers:
  - `_get_zoning` — point-in-polygon parcel centroid vs territorial_zones, LIMIT 1
  - `_get_zouit_overlaps` — все zouit_% layers пересекающиеся с parcel
  - `_get_engineering_nearby` — engineering_structures в 200m, sorted by distance
- `EMPTY_DUMP_RESULT` module-level constant — DRY для no-dump fallback (used
  in get_quarter_dump_data internal + analyze_parcel try/except wrap)

Backend (parcels.py):
- Import EMPTY_DUMP_RESULT + get_quarter_dump_data
- Call wrapped в try/except — если nspd_quarter_dumps недоступна (DB timeout
  / table missing) → EMPTY_DUMP_RESULT fallback вместо 500 (consistent с
  resilience pattern других optional fetches)
- Response gets 4 new fields:
  - nspd_zoning: dict | None (G1 ПЗЗ — zone_code, zone_name, source)
  - nspd_zouit_overlaps: list[dict] (G3 — overlaps в parcel, per ЗОУИТ group)
  - nspd_engineering_nearby: list[dict] (I3 — engineering structures в 200m)
  - nspd_dump: dict (freshness metadata — available, fetched_at_utc, stale,
    harvest_triggered, total_features)

Tests: 13 new в test_quarter_dump_lookup.py (mock-based, no real DB):
- derive_quarter_cad 5 edge cases (3seg, 4seg, 5seg, invalid, whitespace)
- get_quarter_dump no_row → harvest triggered
- stale (>180d) → harvest triggered, stale=True
- harvest_error row → retry harvest triggered
- parcel_wkt=None → metadata only (1 DB call)
- fresh + zoning extraction
- fresh + zouit_overlaps list
- fresh integration: все 4 keys present

47 pre-existing tests still pass.

Code review (code-reviewer pre-push): MINOR, 0 blocking. Applied 2 of 4:
- ✅ #1: try/except wrap around get_quarter_dump_data в analyze_parcel
  (защита от DB unavailability) + DRY через EMPTY_DUMP_RESULT module const
- ✅ #2: removed redundant nspd_zoning.fetched_at_utc (DRY — freshness в
  nspd_dump.fetched_at_utc)
- ⏭ Deferred (acceptable): #3 ad-hoc harvest_quarter retry cooldown для
  harvest_error rows (только при high traffic + persistent NSPD errors);
  #4 raw_props в response — tech debt, убрать вместе с frontend PR

Performance note: 3 spatial queries per analyze adds ~10-50ms on typical
~100-feature quarter. Mitigation if quarters grow dense: materialized
per-layer sub-table (отдельная DB issue).

Closes Sprint 1.1 part of #94. Frontend rendering этих 4 полей — отдельный
PR (next: #112 / #115 / #114).

* fix(site-finder): address PR #116 auto-review M1-M5

M1 (mutation risk): replace EMPTY_DUMP_RESULT direct refs with
_make_empty_result() factory. dict(...) shallow copy left nested
nspd_dump shared by reference across concurrent requests — single
mutation pollutes module sentinel for all subsequent calls. Now
каждый caller gets independent dict.

M2 (O(N) spatial scan): SELECT extended denormalized counts
(territorial_zones_count, zouit_count, engineering_count). Each
spatial helper accepts layer_counts and early-returns when count=0
— skips heavy jsonb_array_elements + ST_Transform/ST_Intersects
scan entirely. Critical для quarters с 2000+ features.

M4 (documentation): _trigger_harvest docstring describes known
burst/no-dedup limitation + TODO Redis SETNX (отдельный PR).

M5 (test fragility): _make_db_mock_with_spatial docstring describes
positional-call contract — db.execute order (0=dump, 1=zoning, 2=zouit,
3=engineering) и зависимость от count-values.

+4 new tests (17 total, all pass):
- test_make_empty_result_returns_independent_copies (mutation safety)
- test_make_empty_result_overrides
- test_early_exit_all_counts_zero_no_spatial_queries
- test_early_exit_partial_counts

Per auto-review on 3068a9c.

* fix(site-finder): rename _make_empty_result → make_empty_result (public) per PR #116 review

M1 residual fix: parcels.py exception path использовал EMPTY_DUMP_RESULT
singleton ref вместо factory. Сейчас readonly access, но нарушает
documented invariant модуля.

Rename `_make_empty_result` → `make_empty_result` (public API), import в
parcels.py, использовать в try/except fallback. Каждый request получает
независимый dict — никаких shared references.

M4 (Redis SETNX dedup) + M5 (test fragility) — deferred per review,
documented в code/issue. Acceptable trade-offs:
- M4: UPSERT idempotency делает данные safe; burst-duplicate task'и тратят
  WAF traffic впустую но не повреждают данные.
- M5: docstring contractually describes positional-call order.

17/17 tests pass. ruff/format clean.

Per auto-review on aef8308.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-13 09:14:19 +03:00
lekss361
f3b0ce5687
feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4 part A) (#111) 
* feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4/4 part A)

Sprint 1.1 item #3 из #94 part 2. Замыкает harvest cycle: Celery task
оборачивает NSPDClient.search_by_quarter (PR #109) + UPSERT в
nspd_quarter_dumps (PR #110). После — analyze_parcel будет читать dumps
вместо on-demand NSPD HTTP в request-цикле (Sprint 1.1 #4, следующий PR).

Backend:
- New `backend/app/workers/tasks/nspd_sync.py`:
  - `harvest_quarter(quarter_cad, region_code=66, include_zouit=True,
    include_risks=False)` — single-quarter harvest. autoretry_for=
    (NspdLiteWafError,) с retry_backoff=True, max_retries=3,
    soft_time_limit=120s. Generic exception → upsert error row с
    harvest_error=str(exc), counts=0 (НЕ re-raise).
  - `harvest_stale_quarters(region_code, max_age_days=90, batch_size=50)` —
    beat fanout: SELECT cad_quarters_geom EXCEPT fresh dumps → apply_async per cad.
  - UPSERT использует ST_Multi(ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON,3857),4326))
    для quarter_geom (schema MultiPolygon strict per database-expert contract).
- `celery_app.py`:
  - Add `nspd_sync` в `include`
  - Beat entry `nspd-harvest-stale-quarters` Mon 04:00 МСК, batch_size=50,
    max_age_days=90
  - `worker_ready` sanity check `SELECT 1 FROM nspd_quarter_dumps LIMIT 0`
    (non-fatal critical log per Bug_Worker_Ready_EarlyReturn_Fixed pattern)
- `admin_scrape.py`:
  - `HarvestQuarterRequest` Pydantic с pattern `^\d+:\d+:\d+$` (quarter only)
  - `POST /api/v1/admin/scrape/nspd/harvest-quarter` manual trigger,
    requires X-Admin-Token, returns task_id.

Tests (`test_nspd_sync.py`): 14 mock-based (no real NSPD/DB):
- happy path: search_by_quarter mock → verify SQL params/structure
- empty quarter → row with quarter_geom=NULL, total=0, no harvest_error
- WAF error → autoretry kicks in (raise re-propagated)
- Generic error → upsert error row
- Fanout: 50 stale → 50 apply_async calls
- Layer names preservation (all 5 zouit + 11 risks if enabled)
- Geometry SRID handling

Code review (code-reviewer pre-push): APPROVE, 0 blocking, 5 minor deferred:
- inline datetime import in error branch (style)
- layers_fetched array literal via string concat (current layer names safe)
- harvest_stale_quarters error rows always re-tried (intended, undocumented)
- test call_args.kwargs fragile (works for current callsite)
- beat collision with refresh-ekb-districts-medians every ~2 years (negligible)

Part of #94. Next: Sprint 1.1 #4 — analyze_parcel reads dump cache.

* fix(nspd-sync): address PR #111 auto-review M1-M5+M7 findings

M1 (runtime bug): remove WHERE region_code from outer cad_quarters_geom SELECT
in harvest_stale_quarters — column does not exist in prod schema
(cad_number, geom, raw_props, fetched_at, source). Would crash on first beat
tick. Inner nspd_quarter_dumps subquery keeps region_code filter unchanged.

M2 (correctness): _upsert_dump теперь делает db.rollback() в except перед
db.close() — prevents PendingRollbackError на следующем session use, если
PostGIS отбракует geometry или JSON cast failure.

M3 (low): worker_ready breadcrumb INSERT тоже теперь имеет rollback —
consistency с остальными session usages в celery_app.py.

M4 (style): layers_fetched — заменил manual string concat
'{' + ','.join(...) + '}' на native Python list. psycopg v3 сам сериализует
list → text[]. Защита от future layer names с запятыми/фигурными скобками.

M5 (style): import datetime as _dt поднят из else branch в top-level imports.

M7 (test cleanup): удалил dead __enter__/__exit__ mock setup в
test_harvest_stale_quarters_fanout / _empty. harvest_stale_quarters не
использует session как context manager — moc'и no-op.

14/14 tests pass. ruff/format clean.

Per auto-review on 9b2a289.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-12 18:45:57 +03:00
lekss361
9ee0b07003
feat(scrapers): search_by_quarter orchestrator + QuarterDump (#94 pt.2/4) (#109) 
Sprint 1.1 item #1 из плана #94 part 2. Foundation для PKK harvest pipeline —
1 vacuum (search) + N layer fetches → comprehensive snapshot всех NSPD данных
в пределах квартала. Базис для G1 #28 ПЗЗ, G3 #30 ЗОУИТ, P2 #46 neighbors,
E1 #51 parcels backfill, #96 ЕГРН помещения.

Backend (nspd_client.py):
- New QuarterDump frozen dataclass (slots=True): quarter + per-layer feature
  lists (parcels/buildings/territorial_zones/red_lines/engineering + zouit
  dict + risks dict) + bbox_3857 + layers_fetched (immutable tuple) +
  fetched_at_utc + total_features property.
- New NSPDClient.search_by_quarter(quarter_cad, include_zouit=True,
  include_risks=False): search → bbox → bulk fetch per layer phase.
  Cost 6/11/22 requests.
- New _geojson_bbox_3857() module-level helper — recursive coord walker.
- Class constants QUARTER_CORE_LAYERS / QUARTER_ZOUIT_LAYERS /
  QUARTER_RISK_LAYERS.

Empty-quarter (NSPD не нашёл cad): quarter=None, bbox=None, all lists empty,
zouit/risks dicts populated с пустыми lists (структурно стабильно),
layers_fetched=('search',).

Tests: +12 tests (31 total, no network).

Code review (code-reviewer pre-push): MINOR, fixed 3 of 5:
- datetime import → module-level
- layers_fetched → tuple[str, ...] (immutable in frozen dataclass)
- docstring clarified empty-quarter semantics
Bonus: ruff UP038 isinstance tuple → union syntax.

Part of #94. Sprint 1.1: 4 PRs total. Next: migration → Celery → integration.

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-12 18:11:30 +03:00
lekss361
bc092b58f1
feat(scrapers): NSPD client foundation (#94) — search + WMS + layers (#98) 
* feat(scrapers): NSPD client foundation (#94) — search_by_cad + WMS + layers

Foundation для G1 #28 ПЗЗ, G2 #29 ВРИ, G3 #30 ЗОУИТ, E1 #51 cad_parcels,
I3 #44 engineering, и поддержки on-demand #93.

Backend (`app/services/scrapers/nspd_client.py`):
- `NSPDClient` с 4 core методами:
  - `search_by_cad(cad, thematic_id)` — REST /api/geoportal/v2/search
    возвращает GeoJSON + ВРИ + land_category + cost_value за 1 запрос
  - `get_feature_info(layer_id, lon, lat, buffer_m=100)` — WMS GetFeatureInfo
    на точке: какие feature'ы layer'а покрывают (lon,lat)
  - `get_features_in_bbox(layer_id, bbox_3857)` — bulk через WMS workaround
    (WFS GetCapabilities → 404, поэтому большой bbox + центральная I/J точка)
  - `list_layers(theme_id)` — каталог слоёв в теме (PKK=1, ARN=665)
- Typed responses: NSPDFeature, NSPDSearchResult, NSPDLayer (frozen dataclasses)
- LAYERS catalog: 32 layer-id с семантическими именами (territorial_zones=875838,
  zouit_engineering=37578, risk_flooding=872205, etc) — TIER 1-6 per #94
- Coordinate helpers: lonlat_to_3857(), bbox_around_point_m()
- Reuse: HEADERS + SSL ctx + fetch_geoportal из existing nspd_lite (WAF-compatible
  urllib trick). Rate limit через rate_ms.
- WAF/Rate-limit: raises NspdLiteWafError на 403/429 — caller backoff.

Tests (`backend/tests/test_nspd_client.py`): 14 unit tests, no network.
- LAYERS catalog sanity (territorial_zones=875838 закрывает G1)
- Coordinate transforms (zero, ЕКБ center, bbox)
- NSPDFeature.from_raw parsing (full / missing fields)
- NSPDSearchResult helpers (empty/first)
- _walk_layer_tree (flat / nested / empty)
- search_by_cad через monkeypatch fetch_geoportal

Также включён micro-fix из PR #95 review (cosmetic):
- parcels.py inline comment "до 25s" → "до 15s" (matched _INLINE_FETCH_WAIT_S)

Scope NOT в этом PR (отдельные follow-up issues, sequential rule):
- Schema migrations (cad_parcels_geom +columns, nspd_territorial_zones,
  nspd_zouit, nspd_red_lines, nspd_risk_zones tables) → отдельный PR
- Celery tasks (sync_territorial_zones_bbox, sync_zouit_*, sync_risk_*) →
  отдельный PR использует client
- Admin endpoints (trigger / status) → отдельный PR
- Замена on-demand fetch на nspd_client → PR after schema ready

Closes part of #94 (foundation only — sub-issues for schema/tasks).

* fix(scrapers): address PR #98 auto-review minor feedback

Backend (nspd_client.py):
1. (line 100+) bbox_around_point_m signature split на multi-line.
2. (lazy imports) import json, time → module-level.
3. (duplicate SSL ctx) Reuse _SSL_CTX из nspd_lite через explicit import.
4. (shape defense) list_layers чекает {"data": [...]} wrapper + non-dict/list garbage → warning + empty.
5. (cad_num docstring) search_by_cad документирует формат + ссылку на validate_cad_format.

Tests: +5 mock тестов (19/19 passed):
- get_feature_info URL builder + parsing
- get_features_in_bbox bbox propagation
- list_layers_walks_tree_response
- list_layers_handles_data_wrapper
- list_layers_handles_garbage_response

Backend (parcels.py): docstring '~25с' → '~15с'.

Per auto-review on be14388.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-12 09:47:45 +03:00
lekss361
b95559eac9 init 2026-04-25 13:45:19 +03:00