Commit graph

132 commits

Author SHA1 Message Date
e673811a20 feat(parcels): layout signature schemas + extractor (#113 PR A) (#193)
All checks were successful
Deploy / changes (push) Successful in 4s
Deploy / build-backend (push) Successful in 1m45s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 3m1s
Deploy / deploy (push) Successful in 49s
2026-05-16 07:52:34 +00:00
4deb1b0284 feat(parcels): connection-points endpoint (Forgejo #115 Phase 1 backend) (#190)
Some checks failed
Deploy / changes (push) Successful in 5s
Deploy / build-frontend (push) Failing after 0s
Deploy / build-worker (push) Successful in 5m11s
Deploy / deploy (push) Failing after 0s
Deploy / build-backend (push) Successful in 3m44s
2026-05-16 05:58:27 +00:00
e561df1b55 feat(parcels): competitors endpoint (Forgejo #112 Phase 1 backend) (#191)
Some checks failed
CI / backend (push) Successful in 1m16s
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Failing after 12s
Deploy / build-worker (push) Failing after 12s
Deploy / build-frontend (push) Failing after 0s
Deploy / deploy (push) Failing after 0s
CI / frontend (push) Successful in 2m17s
2026-05-15 22:21:05 +00:00
50547dcbb3 test(deploy): validate Forgejo Actions end-to-end flow (#189)
Some checks failed
CI / backend (push) Successful in 1m40s
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Failing after 11s
Deploy / build-worker (push) Failing after 11s
CI / frontend (push) Successful in 2m13s
Deploy / build-frontend (push) Failing after 12s
Deploy / deploy (push) Failing after 0s
2026-05-15 21:55:04 +00:00
lekss361
4d492ebc50
perf(cadastre): freshness skip + grid-walk heartbeat + log noise reduction (#187)
* perf(cadastre): freshness skip + grid-walk heartbeat + reduce WMS log noise

Three operational improvements after ekb_full v3 completed at 99.1%:

Fix H — per-quarter freshness skip (scrape_cadastre.py):
- enqueue_cadastre_harvest queries cad_quarter_stats.fetched_at per quarter
- Skip quarters where fetched_at >= NOW() - skip_fresh_hours (default 1h)
- extra_config.skip_fresh_hours=0 disables skip (full re-fetch)
- Saves ~30k NSPD calls on repeated runs of same scope

Fix I — heartbeat in grid-walk loop (bulk_harvest.py:_grid_walk_category):
- 225 cells x 70-300ms = 15-75s heartbeat silence
- Pass update_progress callback to _grid_walk_category
- Call every heartbeat_every (50) cells with phase progress info
- Prevents cleanup_zombies false-positive cancellation

Fix J — NSPD WMS HTTP 500 noise to debug (bulk_harvest.py:_grid_walk_category):
- NSPD WMS endpoint frequently returns HTTP 500 (server-side issue)
- Was: logger.warning each, 200+ warnings per quarter
- Now: logger.debug (visible via --log-level=DEBUG)

40 tests passing.

* fix(cadastre): parse NSPD DD.MM.YYYY dates (Fix K — job 18 root cause)

Job 18 (manual_list recovery 11 quarters) crashed every quarter with:
  psycopg.errors.DatetimeFieldOverflow: date/time field value out of range: '13.03.2023'

NSPD returns registration_date in mixed formats:
- DD.MM.YYYY for Сооружения / ОНС / ЕНК / ЗОУИТ
- YYYY-MM-DD ISO for ЗУ / Здания
- YYYY-MM-DD HH:MM:SS with time

PG CAST(:date AS date) accepts ISO only — DD.MM.YYYY crashes.

Add _parse_nspd_date helper: handles both formats + None/empty.
Apply in upsert_construction, upsert_onc, upsert_enk, upsert_zouit, upsert_parcel
for registration_date / build_record_registration_date / legal_act_date.

3 new unit tests for the helper (43 tests total).

* fix(cadastre): address bot blockers — pivot Fix H to raw_targets + tests

Bot review on PR #187 caught:
1. cadastre_jobs has no extra_config column → Fix H crashed on KeyError
2. Missing tests for Fix H + Fix I
3. _parse_nspd_date didn't validate date ranges

Changes:
- Pivot Fix H: read skip_fresh_hours from raw_targets JSONB (existing column),
  no schema migration needed
- _parse_nspd_date: validate via date.fromisoformat() — rejects 2023-13-45
- Add 3 tests for Fix H in backend/tests/workers/test_scrape_cadastre.py
- Add 2 tests for Fix I heartbeat callbacks in test_cadastre_bulk.py
- Add 1 test for parse_nspd_date range validation (was: only format)

49 tests passing.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 23:02:37 +03:00
lekss361
d104222643
fix(cadastre,nspd_sync): SQL syntax bugs surfaced by ekb_full v2 (#186)
Three distinct SQL syntax errors caught from worker logs:

1. **upsert_zouit** (bulk_harvest.py:940) — trailing comma after 4326
   in ST_Transform(..., 3857), 4326,) → 'syntax error at or near ")"'.
   Hit on 8+ quarters in 0108 block (66:41:0108055/61/69/77 etc).
   Likely introduced by ruff-format auto-comma on closing paren.

2. **nspd_sync** (nspd_sync.py:120) — `:fetched_at_utc::timestamptz`
   psycopg double-colon trap → 'syntax error at or near ":"'.
   Crashed legacy on-demand harvest_quarter called from analyze.

3. **job_settings** (job_settings.py:224) — `:extra_config::jsonb`
   same double-colon trap in admin settings UPDATE.

All converted to CAST(:x AS type) pattern (psycopg v3 safe).

40 tests still passing.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 19:18:43 +03:00
lekss361
5f8df655cc
fix(cadastre): cad_parcels.geom Polygon -> MultiPolygon (migration 93) (#185)
NSPD returns MultiPolygon for Многоконтурный участок (e.g. 66:41:0105017:4)
which crashed upsert_parcel — schema was strict POLYGON.

Migration 93: ALTER COLUMN geom Polygon -> MultiPolygon USING ST_Multi(geom),
DROP+RECREATE GIST index + back-compat VIEW cad_parcels_geom.

upsert_parcel SQL wraps ST_Transform in ST_Multi() — coerces Polygon to
MultiPolygon. NSPD returns both types (simple + multi-contour parcels).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 18:52:38 +03:00
lekss361
e3359c4f03
fix(cadastre): filter Point/LineString geometry in upsert_parcel/zouit (#184)
cad_parcels.geom is geometry(Polygon, 4326) — strict schema. NSPD
occasionally returns Point geometry for parcels without detailed
boundary, causing INSERT failure:
  psycopg.errors.InvalidParameterValue:
  Geometry type (Point) does not match column type (Polygon)

This killed pilot v8 at quarter 66:41:0104002 — worker autoretry
exhausted, job hung at 25/50 (heartbeat stale).

Fix: filter geometry.type at Python level in upsert_parcel + upsert_zouit
(same pattern as upsert_quarter_geom_from_feature). Non-Polygon
geometry → geom=NULL, raw_props preserved.

Tables with permissive GEOMETRY schema (cad_buildings, cad_constructions,
cad_enk, cad_oncs) unaffected — they accept any geometry type.

Tests: 3 new (Point parcel → geom=NULL, Polygon happy path, LineString zouit).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 18:37:29 +03:00
lekss361
e695bca1e4
fix(cadastre): remove shared phase_state early-exit (race condition) (#183)
cadastre_jobs.phase_state is shared between ALL parallel Celery workers
of the same job. The early-exit triggered as soon as the FIRST worker
wrote phase=done via progress_cb — all subsequent workers (reading same
shared row) bailed without work.

Effect: pilot v7 made 5 NSPD requests for 50 quarters (=0.1/quarter).
Full ekb_full job #8: 4950 req / 2408 quarters = 2/quarter — 95% workers
early-exited without doing snapshot phase.

Root cause of 1.6% parcels coverage. Fix A/B/C only partially helped
because most workers never reached Phase 1/1.5.

Solution: delete the early-exit. Idempotency is guaranteed by
ON CONFLICT DO UPDATE in every upsert_* — re-enqueued tasks
write same rows without duplicates.

Closes part of #168 (root-cause fix for race condition revealed by #182 metrics).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 17:41:27 +03:00
lekss361
057f8891dc
fix(cadastre): count snapshot+per-cat requests in cadastre_jobs.requests_count (#182)
scrape_cadastre.py previously incremented requests_count only by
grid_walk_requests, so snapshot-only quarters (and the entire ekb_full
job after PR #179 skips grid-walk on broken geom) reported 0 NSPD
requests — misleading for observability.

Add HarvestResult.snapshot_requests counter (Phase 1 search + Phase 1.5
per-cat probes) and a total_requests property. Update Celery task to
write total_requests into cadastre_jobs.requests_count.

Tests added for the new property + Phase 1 snapshot_requests=1 assertion.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 17:17:01 +03:00
lekss361
905ef443fc
fix(cadastre): auto-heal cad_quarters_geom from CAT_QUARTER_STATS feature (#181)
72% of EKB quarters (1735/2408) have broken micro-precision geom in
cad_quarters_geom. NSPD returns valid quarter polygon in CAT_QUARTER_STATS
(36381) feature but Phase 4 was only consuming stats.

Add upsert_quarter_geom_from_feature: CTE-based UPDATE that transforms
3857 polygon to 4326 and writes only when existing geom is NULL or broken
(bbox width outside 100-10000m), verified via SQL-side sanity check.
Wire into Phase 4 with begin_nested savepoint: malformed NSPD GeoJSON
(self-intersecting ring) rollbacks only geom UPDATE, preserves
upsert_quarter_stats.

Closes part of #168 (follow-up data quality fix for #179).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 17:16:39 +03:00
lekss361
d5b57ccbf1
feat(cadastre): per-category NSPD snapshot — Phase 1.5 (ZOUIT/ENK) (#180)
Switch base search_by_quarter from thematicSearchId=1 to thematic=1
(includes ZOUIT in features). Add optional category_id param that routes
ZOUIT (36940/469039/469040/469042) -> thematicSearchId=5,
ENK (39663) -> thematicSearchId=15.

Add Phase 1.5 in harvest_quarter: for each PER_CATEGORY_PROBE_CATS
category present in meta_counts (meta_total>0), run dedicated
per-category search call. Skips if category in GRID_WALK_CATS and
bbox is valid (grid-walk gives better coverage).

Curl-probe confirmed NSPD API caps at 20/category with no pagination;
categoryId param is server-ignored; thematic=1 strictly better than
thematicSearchId=1 (includes ZOUIT).

Closes part of #168 (ZOUIT/ENK coverage gap).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 16:53:49 +03:00
lekss361
3beb75d68a
fix(cadastre): skip grid-walk for broken quarter geometries (#168) (#179)
* fix(cadastre): skip grid-walk for broken quarter geometries (#168)

Pilot v5 (50 quarters) разоблачил что cad_quarters_geom имеет 1735/2408 (72%)
ЕКБ rows с broken micro-precision geom (width ~0.01м instead of 200-4000м).
Causes:
- grid-walk WMS receives bbox ~1mm × 1mm → 500 ServiceException на all 450 req
- One quarter hung 2+ min (silent failure in async loop)

Real fix: re-import cad_quarters_geom geometry from NSPD (category 36381).
Tracked в follow-up issue.

Quick fix (this PR): sanity-check quarter_bbox_3857 — return None если width
ИЛИ height not in [100m, 10000m]. Bulk_harvest_quarter then skips grid-walk
для broken quarters, только snapshot phase запускается (20 per cat).

After this fix:
- All 50 pilot quarters получают snapshot (parcels/buildings/etc up to 20 each)
- Grid-walk runs only для 673/2408 healthy quarters → no 500 cascades, no hangs

* test(cadastre): add quarter_bbox_3857 boundary tests (PR #179 bot blocker)

- healthy bbox 750m -> returns tuple
- width <100m (broken micro-precision) -> None
- width >10000m (oversized) -> None
- height <100m (symmetry) -> None
- row missing (quarter not in cad_quarters_geom) -> None

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 16:25:06 +03:00
lekss361
6dfff5e046
fix(cadastre): CAST :geom AS text — psycopg AmbiguousParameter (#168) (#178)
Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 16:02:03 +03:00
lekss361
99e2210919
fix(cadastre): NSPD response parse — properties.category + top-level meta (#168) (#177)
* fix(cadastre): NSPD response parse — properties.category + top-level meta (#168)

* fix(cadastre): test fixtures use real NSPD shape — properties.category + top-level meta (#168)

Bot review of #177 flagged that fixtures still used legacy properties.categoryId
+ data.meta — meaning CI exercised only fallback branches, not primary paths
where the actual bug lived. Pattern of 3 consecutive "live-only" parse fixes
(#175 verify, #176 headers, #177 parse) confirmed need for test coverage.

Changes:
- test_nspd_bulk_client.py: sample_quarter_response → properties.category +
  meta moved to top-level (real NSPD shape verified live)
- test_cadastre_bulk.py: 7 fixtures categoryId → category (regex replace)
- test_nspd_bulk_feature_parse_basic: primary path now exercised

Plus schema hardening per bot review:
- NSPDBulkFeature.category_id: `is not None` check (not truthy `or`)
  to avoid edge case category=0; int() wrapped in try/except so
  non-numeric ID (e.g. "ЗУ") doesn't crash upsert_features loop.

35/35 tests pass locally.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 15:40:01 +03:00
lekss361
608490fbba
fix(parcels): _parse_floors handle int (post-migration #169) (#176)
* fix(parcels): _parse_floors handle int (post-migration #169 schema change)

After PR #169 cad_buildings schema migration, `floors` column is INT
(was TEXT in legacy schema). Existing `_parse_floors(r.get("floors"))`
call in analyze_parcel → _neighbors_summary crashes with:
  AttributeError: 'int' object has no attribute 'strip'

Fix: type union str | int | None. If int → return directly (no strip).
Preserve TEXT range parsing ("5-7" → 7) for backwards-compat with
any legacy data still in cad_buildings_old_apr26.

* test(smoke): production smoke tests for post-deploy regressions (#168)

Add tests/smoke/test_prod_smoke.py covering known regression surfaces.
Marks: prod_smoke + slow. Env: PROD_SMOKE_BASE_URL, PROD_SMOKE_ADMIN_TOKEN.
Run manually: cd backend && uv run pytest tests/smoke/ -m prod_smoke -v

* fix(cadastre): exact-match headers to nspd_lite to bypass NSPD WAF (#168)

Pilot v2 (job_id=2) failed 50/50 with HTTP 403 WAF block. Comparing
nspd_bulk_client.DEFAULT_HEADERS vs legacy nspd_lite.HEADERS (which works
on VPS IP since April 2026):

  PascalCase keys → lowercase keys
  Chrome/148 UA → Chrome/144 UA
  No cache-control / pragma → "no-cache" both
  accept-language ru first → en first
  No origin → "https://nspd.gov.ru"
  referer "/map" → "/map?thematic=PKK"

NSPD WAF (BotShield-class) likely fingerprints на header order + values
combined with TLS fingerprint. Matching legacy exactly minimizes deltas.

Test plan: retry pilot job after deploy, expect 0 WAF blocks for first
5 quarters.

* fix(test): exclude prod_smoke tests by default (#168)

CI ran tests/smoke/test_prod_smoke.py and they hit production
https://gendsgn.ru/api/v1/parcels/.../analyze which currently returns 500
(parse_floors regression — exactly what this PR fixes). Catch-22: PR can't
merge because smoke tests fail against pre-merge prod.

Fix: add `addopts = ["-m", "not prod_smoke"]` so default pytest excludes
them. Run manually post-deploy with: pytest -m prod_smoke -v

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 15:08:16 +03:00
lekss361
8dee9546ae
fix(cadastre): NSPDBulkClient verify=False for NSPD SSL chain (#168 hotfix) (#175)
Pilot run #1 failed on all 50 quarters with:
  NspdBulkError: [SSL: CERTIFICATE_VERIFY_FAILED] self-signed certificate
  in certificate chain

NSPD prod chain on Beget VPS contains an internal/self-signed CA →
httpx default verify=True trips. Reuse pattern from legacy
app.services.scrapers.nspd_lite (uses ssl._create_unverified_context()
for same reason since April 2026).

Risk profile: NSPD public gov data, no sensitive payload outbound,
admin token не уходит к NSPD endpoints. MITM risk minimal.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 14:50:21 +03:00
lekss361
5da66bbc93
feat(cadastre): bulk_harvest worker + grid-walker + admin API (#168 PR3/5) (#171)
* feat(cadastre): bulk_harvest_quarter Celery task + grid-walker + saga state (#168 PR3/5)

Add bulk cadastre harvest pipeline:
- services/cadastre/bulk_harvest.py: async harvest_quarter() orchestrator (4 phases)
  + 7 upsert helpers (parcels/buildings/constructions/oncs/enks/zouit/quarter_stats)
  using CAST(:x AS jsonb) pattern, begin_nested() SAVEPOINT per grid-walk upsert
- services/cadastre/grid_geometry.py: quarter_bbox_3857 (PostGIS ST_Extent)
  + generate_grid_click_points (15x15 = 225 sub-bbox grid)
- workers/tasks/scrape_cadastre.py: bulk_harvest_quarter_task (acks_late=True,
  dont_autoretry_for NspdBulkWafError), enqueue_cadastre_harvest, cleanup_zombies
- api/v1/admin_cadastre.py: 5 endpoints behind AdminTokenAuth — create/list/get/cancel/resume
- Tests: 13 service unit + 8 API tests

* fixup(cadastre): drop importorskip (PR2 merged) — imports now top-level (#168 PR3)

* fixup(cadastre): tile_size unified param + register slow marker (#168 PR3)

Blocker #1: rename tile_width/tile_height → tile_size in generate_grid_click_points.
Callers in bulk_harvest.py and test_cadastre_bulk.py used tile_size; def had
tile_width+tile_height → TypeError at runtime.

Blocker #2 (10 failures in test_admin_cadastre.py) was side-effect of #1:
TypeError at import chain (app.main → admin_cadastre → bulk_harvest)
broke FastAPI app load → dependency_overrides AttributeError. Now resolved.

Also register `slow` pytest marker in pyproject.toml to suppress
PytestUnknownMarkWarning.

* fixup(cadastre): use importlib.util.find_spec instead of import (#168 PR3)

Root cause Blocker #2: `import app.workers.tasks.scrape_cadastre` at module
top-level REBOUND `app` from FastAPI instance (line 14) to the Python package.
All subsequent `app.dependency_overrides[get_db] = ...` failed with
AttributeError because `app` was now the namespace module.

Fix: probe module availability with importlib.util.find_spec — does not
import or rebind names. FastAPI `app` instance stays intact.

All 28 cadastre tests pass locally.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 13:31:32 +03:00
lekss361
b7889d113b
feat(cadastre): NSPDBulkClient + schemas for bulk ingest (#168 PR2/5) (#170)
Add async NSPD client (nspd_bulk_client.py) with 3 new methods:
- search_by_quarter: REST /api/geoportal/v2/search/geoportal — snapshot 20 per cat + meta totals
- wms_feature_info: WMS GetFeatureInfo для grid-walk discovery
- list_objects_in_building: tab-group-data (Q3 deferred — код ready)

Rate limit: asyncio.Semaphore(3) + 0.05s jitter.
429 → exponential backoff 2→4→8s, max 3 retries.
403 (WAF) → raise immediately, no retry.

Pydantic schemas (nspd_bulk.py): NSPDBulkFeature, QuarterSnapshot, ObjectsListing
with extra='allow' to preserve unknown fields → raw_props JSONB downstream.

15 unit tests + 4 e2e (real NSPD, skipped in CI by default).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 12:50:41 +03:00
lekss361
d7fbaa0528
feat(velocity): per-room-bucket breakdown in VelocityResult (#163)
* fix(velocity): :window_interval::interval cast syntax — same bug as PR #152

SQLAlchemy text() parser confuses ':window_interval' (named param) followed
by '::interval' (cast operator) — exactly как было в PR #152 с :weights::jsonb.

Result: psycopg видит literal ':window_interval' в SQL → syntax error →
exception caught by velocity catch → return None → UI shows null.

## Fix

':window_interval::interval' → 'CAST(:window_interval AS interval)' (2 мест:
sales_rows query + _get_ekb_median percentile).

Pre-push code-reviewer должен был catch — добавим в feedback что для каждого
text() SQL grep ':[a-z]*::' before push.

Refs: PR #158 deploy verify, e2e velocity = null root cause

* feat(velocity): add per-room-bucket breakdown to VelocityResult

Add third SQL query (bucket_rows) aggregating deals_total_count by
room_bucket ('студия','1','2','3','4+') across mapped competitors.

New fields:
- VelocityResult.by_room_bucket: aggregate {units, sqm, complexes_count}
- sample_competitors[].by_room_bucket: {bucket: units}

Bucket query wrapped в SAVEPOINT — failure degrades gracefully к empty
dict без abort outer tx.

CAST(:window_interval AS interval) pattern per PR #160 lesson.

Test coverage: 3 new tests (aggregation, empty fallback, sample entries).

Closes part of #161

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 10:09:03 +03:00
lekss361
30d78c5030
fix(pipeline): enrich obj_class via objective_lots COALESCE fallback (#165)
domrf_kn_objects.obj_class всегда NULL (Bug_Kn_API_Obj_Class_Always_Null_OPEN).
Pipeline 24mo UI показывал 'Не указан: 4075 квартир'.

Fix: SQL COALESCE с correlated subquery через objective_complex_mapping →
objective_lots.class. Mapped complexes (114/4548) теперь резолвят корректный
класс — комфорт/стандарт/бизнес/премиум.

## Coverage impact (verified via MCP)

Pre-fix: 0 of 1316 planned ЖК имели obj_class
Post-fix: 339 of 1316 (25.8%) — 249 комфорт, 45 стандарт, 42 бизнес

Не-mapped objects остаются 'Не указан' (upstream kn-scraper fix всё ещё open).

Closes #162

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 10:07:52 +03:00
lekss361
46584d7c75
fix(velocity): :window_interval::interval cast syntax — same bug as PR #152 (#160)
SQLAlchemy text() parser confuses ':window_interval' (named param) followed
by '::interval' (cast operator) — exactly как было в PR #152 с :weights::jsonb.

Result: psycopg видит literal ':window_interval' в SQL → syntax error →
exception caught by velocity catch → return None → UI shows null.

## Fix

':window_interval::interval' → 'CAST(:window_interval AS interval)' (2 мест:
sales_rows query + _get_ekb_median percentile).

Pre-push code-reviewer должен был catch — добавим в feedback что для каждого
text() SQL grep ':[a-z]*::' before push.

Refs: PR #158 deploy verify, e2e velocity = null root cause

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 09:46:01 +03:00
lekss361
8fa1d005fd
feat(velocity): migrate D2 sales source → objective_corpus_room_month (#158)
* feat(velocity): migrate D2 sales source from stale sale_graph to objective_corpus_room_month

Closes #156

* fix(velocity): cross-stack contract + NULL safety per pre-push review

3 pre-push code-reviewer findings fixed:

1. TS union extended ('objective'|'sale_graph'|'rosreestr_fallback')
   + UI conditions handle both objective и sale_graph как valid sources.
2. COALESCE(deals_total_vol_m2, deals_total_count * 45.0) — NULL safety
   for DKP-only rows (vol_m2 nullable, count > 0).
3. room_bucket parking filter — verified false positive (все 5 buckets
   apartments: студия/1/2/3/4+).

Refs: PR #157 pre-push code-reviewer

* fix(types): add 'objective' to velocity_source union (cherry-pick miss)

Cherry-pick от закрытого PR #157 потерял этот файл. Frontend type-check fails
с 'no overlap' error. Restoring.

Refs: PR #158 frontend CI failure

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 09:34:00 +03:00
lekss361
c39b53cf53
fix(site-finder): cascade InFailedSqlTransaction — velocity internal rollback (#155)
* fix(site-finder): db.rollback() в velocity catches + SAVEPOINT zoning/success_rec

velocity.py 3 internal try/except возвращали None БЕЗ db.rollback(). PR #154
SAVEPOINT не помог — exception не propagates из velocity. Postgres tx
aborted внутри, cascade InFailedSqlTransaction на следующих queries.

Fix:
- velocity.py: db.rollback() в 3 catch блоках
- parcels.py: SAVEPOINT обёртки для zoning + success_recommendation

Refs: user reports 2026-05-15 cascade 500

* fix(velocity): SAVEPOINT per query (not db.rollback) — correct SQLAlchemy 2.0 pattern

Per PR #155 bot review — мой db.rollback() в catches был НЕПРАВИЛЬНЫЙ.

Session.rollback() = ENTIRE outer tx (не savepoint). Outer SAVEPOINT context
становится orphaned → InvalidRequestError на __exit__.

Correct fix: wrap каждый db.execute в свой with db.begin_nested():
- Failure → __exit__ propagates exception → SAVEPOINT rolls back
- Outer tx остаётся clean
- velocity returns None gracefully → caller продолжает

Все 3 SQL queries в velocity.py теперь SAVEPOINT-wrapped:
- compute_velocity: competitor_query
- compute_velocity: sales_rows query
- _get_ekb_median: median query

db.rollback() removed из всех 3 catches.

Refs: PR #155 bot review

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 08:52:38 +03:00
lekss361
b13c4be0ed
fix(parcels): SAVEPOINT around velocity compute → не abort outer tx (#154)
Production POST /parcels/{cad}/analyze падает с 500:
  sqlalchemy.exc.InternalError: (psycopg.errors.InFailedSqlTransaction)
  current transaction is aborted, commands ignored until end of transaction block

## Root cause

`compute_velocity()` SQL fails (probably на cad's без conkurrentов / sparse
sale_graph data) → exception caught в try/except → НО db.rollback() отсутствует
→ transaction остаётся в aborted state.

Следующая query (_geotech_risk на line 828) пытается выполниться → крашится
с InFailedSqlTransaction.

## Fix

Wrap velocity в `with db.begin_nested()` — SAVEPOINT pattern (consistent
with PR #124 pzz_loader fix). Failure внутри savepoint:
- Rollbacks ТОЛЬКО savepoint
- Outer transaction остаётся clean
- Subsequent queries (_geotech_risk и пр.) работают

Pattern matches feedback_subagent_delegation audit recommendation для
in-loop / per-section exception handling.

## Impact

POST /parcels/{cad}/analyze больше не 500 при velocity failure. Возвращает
`velocity: null` + остальные fields normal.

Refs: user report 2026-05-15 InFailedSqlTransaction

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 08:29:24 +03:00
lekss361
af39e95f83
fix(site-finder): SQLAlchemy syntax error :weights::jsonb → CAST(:weights AS jsonb) (#152)
Production POST /weight-profiles → 500 SyntaxError.

## Root cause

SQLAlchemy text() parser трактует ':weights' followed by '::jsonb' двусмысленно:
- ':weights' — named param binding (должен стать %(weights)s)
- '::jsonb' — PG cast operator

Parser НЕ распознаёт :weights как параметр когда сразу следует :: cast.
Result: psycopg видит literal ':weights' в SQL → syntax error.

```
LINE 5: ($1, $2, :weights::jsonb, $3, $4)
                   ^
parameters: {user_id, profile_name, is_default, description}
            ↑ note: 'weights' missing — SQLAlchemy skipped it
```

## Fix

Replace ambiguous :weights::jsonb с CAST(:weights AS jsonb):
- _INSERT (line 102)
- _UPDATE dynamic 'weights = :weights::jsonb' (line 273)

CAST() syntax не имеет ambiguity, SQLAlchemy корректно bind'ит :weights.

## Affected endpoints

- POST /api/v1/admin/site-finder/weight-profiles (create)
- PUT /api/v1/admin/site-finder/weight-profiles/{id} (update with weights)

Refs: user report 2026-05-15

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 08:12:18 +03:00
lekss361
e93bb6121e
feat(site-finder): D2 velocity-score from domrf_kn_sale_graph (#34 sub-PR 1/2) (#146)
compute_velocity service queries competitor sales в радиусе 3км:
- ST_MakePoint(longitude, latitude) — domrf_kn_objects не имеет geom column
- JOIN domrf_kn_sale_graph за 6 мес (area_sq primary, realised*45 fallback)
- Normalize vs ЕКБ-wide median → velocity_score 0..1
- confidence: high/medium/low (competitors_count + months_observed)
- Top 5 sample competitors для UI

Integration: analyze_parcel.response['velocity'] top-level field.

Schema corrections vs spec:
- obj_name → comm_name
- region_code → region_cd
- contracted (INT) → area_sq (м²)

Tests: 102/102 pass.
Vault: Module_Velocity_Service.md NEW.

Closes #144 (sub-PR 2 frontend закроет #34)

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 01:24:53 +03:00
lekss361
76a88b2827
feat(site-finder): gate verdict aggregator — can-build-MKD signal (#32 G5 sub-PR 1/2) (#142)
Pure-function aggregator collapses nspd_zoning / nspd_zouit_overlaps /
nspd_engineering_nearby / nspd_dump в один GateVerdict TypedDict.

Logic:
- ПЗЗ не Ж-* → BLOCKER
- ЗОУИТ sub=17 (инжен. охранная) → BLOCKER
- Другие ЗОУИТ → WARNING
- Нет инжен. сетей в 200м → WARNING
- nspd_dump stale → source: nspd_dump_partial
- Нет dump → can_build_mkd: 'unknown'

Integration: новое поле gate_verdict в analyze_parcel response.

Tests: 15/15 pass (mock-based).

Vault: code/modules/Module_Gate_Verdict.md NEW.

Closes #140 (sub-PR 2 frontend закроет #32)

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 00:59:53 +03:00
lekss361
7250aa6187
feat(site-finder): weight profiles endpoints + analyze integration (#114 sub-PR 3/4) (#138)
Admin CRUD router + non-breaking analyze_parcel integration. 46 tests pass.

Refs: #114

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 00:40:06 +03:00
lekss361
5aea78c2d8
feat(site-finder): weight profiles service — Pydantic + CRUD (#114 sub-PR 2/4) (#137)
Per #114 sub-PR 2/4. Pydantic v2 models + CRUD service. 14 passing tests.

API: list/get/get_default/create/update/delete_profile + resolve_weights
fallback to system defaults.

Validation: ALLOWED_CATEGORIES guard + weight bounds [-2.0, 3.0].
Default-uniqueness в одной transaction (job_settings.py pattern).

Vault: code/modules/Module_Weight_Profiles_Service.md.

Refs: #114

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 00:29:37 +03:00
lekss361
6a256bfc1f
refactor(workers): split celery_app.py god-object into 3 modules (#135)
Per audit batch #127 P3 god objects (issue #134).

Split layout:
- celery_app.py: 270 → 35 lines (config only)
- beat_schedule.py: 207 lines NEW (DB-driven + fallback + hardcoded)
- lifecycle.py: 168 lines NEW (worker_process_init + worker_ready handlers)

No business logic changes — move-only structural refactor. Public API
(celery_app.conf.beat_schedule) unchanged.

Verify:
- beat keys: kn-region-66, objective_sync, refresh-ekb-districts-medians,
  nspd-geo-zombie-cleanup, poi-sync-weekly, noise-sync-weekly
- ruff + ruff-format clean
- AST parse OK

Vault: Module_Beat_Schedule + Module_Worker_Lifecycle NEW;
Module_Celery_App updated.

Closes #134
Refs: #127

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 00:09:38 +03:00
lekss361
389fd31fad
refactor(admin): extract _check_token into shared AdminTokenAuth dep (#133)
Per audit batch #127 P2 hygiene (issue #128).

## Problem

Identical `_check_token` function в 3 admin files:
- backend/app/api/v1/admin_scrape.py:37-44
- backend/app/api/v1/admin_leads.py:26-33
- backend/app/api/v1/admin_jobs.py:24-31

## Fix

New `backend/app/core/deps.py` с `verify_admin_token` + Annotated alias:
```python
AdminTokenAuth = Annotated[None, Depends(verify_admin_token)]
```

Заменено в 27 endpoints (18 admin_scrape + 6 admin_leads + 3 admin_jobs):
- Removed local `_check_token` definitions
- `_: None = Depends(_check_token)` → `_: AdminTokenAuth`
- Removed unused imports (`Header`, `HTTPException`, `settings` где не используется)

## Semantics preserved

Все 3 original implementations были identical (503 if env missing + 401 if
mismatch, same header `X-Admin-Token`, same env `SCRAPE_ADMIN_TOKEN`).

Минор: 503 detail text unified to `"admin disabled — set SCRAPE_ADMIN_TOKEN"`
(в оригинале три разные строки — `"admin scrape disabled..."` etc).
Cosmetic — detail в HTTP 503 не парсится клиентским кодом.

## Tests

`uv run pytest tests/ -x -k admin` → 64/64 passed.

## Vault

`code/patterns/Pattern_Admin_Token_Dependency_May14.md` — created.

Closes #128
Refs: #127

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:45:44 +03:00
lekss361
a3e7d21e3a
fix(pzz_loader): replace bare db.rollback() with SAVEPOINT per-row (#124)
Per code review audit (May 14): transaction correctness bug в
`sync_pzz_zones_to_db()` — inner `db.rollback()` в loop сбрасывал outer
transaction на per-row failures → counts inserted/updated НЕ matched DB state.

## Fix

Wrap UPSERT в loop через `db.begin_nested()` (SAVEPOINT):
- Failure одной row → rollback только её savepoint, outer tx alive
- Successfully inserted rows accumulate в outer tx → commit final
- Outer try/except добавлен для unexpected error → rollback + log + raise
- `finally: db.close()` preserved

Pattern consistent с `domrf_kn.py` (где SAVEPOINT уже работает).

## Preserved

- Function signature + return shape {fetched, inserted, updated, skipped} — unchanged
- Logging contracts
- skipped count теперь корректно включает DB-failure rows (consistent with prior behavior)

## Tests

- No test_pzz_loader.py — skipped
- ruff + AST passed

## Vault

`fixes/Bug_Pzz_Loader_Missing_Savepoint_May14.md` — created (status: resolved).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:29:23 +03:00
lekss361
8dfab7613d
fix(nspd-geo): replace f-string bulk INSERT with parametrized batch execute (#123)
Per code review audit (May 14): real SQL injection vector в `enqueue_geo_job`.
Manual `chr(39)*2` quote escape НЕ injection-safe — не защищает от backslash
escape, Unicode quote tricks. Если cad_num когда-либо придёт из user input —
exploitable.

## Change

`backend/app/workers/tasks/nspd_geo.py:298-310` — заменил f-string VALUES
concatenation на SQLAlchemy `text(...)` + list of dicts (psycopg v3 handles
batch executemany под капотом).

Pattern consistent с остальными INSERT'ами в этом же файле (_log,
_start_job, _save_quarter, _save_parcel, _save_building).

## Preserved

- ON CONFLICT (job_id, cad_num, thematic_id) DO NOTHING — verbatim
- Return value (int(job_id))
- Surrounding tx/commit/close logic
- DB schema, function signature, caller interface — unchanged

## Tests

- No test_nspd_geo.py существует — skipped
- ruff check passed

## Vault

`fixes/Bug_Nspd_Geo_Sql_Injection_May14.md` — created (status: resolved).

No backward compat / migration concerns — pure in-place query rewrite.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:29:12 +03:00
lekss361
f5794173de
fix(celery): disable nspd-harvest-stale-quarters beat (Bug post-mortem #1) (#121)
Per fixes/Bug_NSPD_WMS_NotBulk_2026_May14 action item #1: отключить beat
schedule `nspd-harvest-stale-quarters` до Sprint 2 fix (grid sampling rewrite).

## Why

harvest_quarter в текущей реализации пишет почти пустые dumps в
nspd_quarter_dumps из-за single-pixel WMS GetFeatureInfo bug (детали в
vault post-mortem). Beat запуск Mon 04:00 МСК (next: 2026-05-18 04:00 МСК):

- Потратит rate-limit budget на ~12k EKB кварталов
- Заполнит nspd_quarter_dumps мусорными rows
- analyze_parcel начнёт отдавать пустые nspd_zoning/nspd_zouit_overlaps/
  nspd_engineering_nearby для всех пользователей (через quarter_dump_lookup)

## What changed

`backend/app/workers/celery_app.py` — закомментирован `schedule[
"nspd-harvest-stale-quarters"]` блок (~10 строк) с inline-комментом
explanation + ссылкой на vault post-mortem.

## What НЕ changed

- `backend/app/workers/tasks/nspd_sync.py` — task code (harvest_quarter,
  harvest_stale_quarters) остаётся, можно re-enable после Sprint 2 fix
- Admin endpoint для manual trigger остаётся работающим
- Все остальные beat tasks (poi-sync, noise-sync, zombie-cleanup,
  refresh-ekb-districts-medians, kn-region-*) — не тронуты

## Vault

- fixes/Bug_NSPD_WMS_NotBulk_2026_May14.md — status `identified` → `mitigated`
  (уже обновлено через vault audit pass)

## Re-enable

После Sprint 2 grid-sampling rewrite (см. Bug post-mortem → Sprint 2
fix-strategy) — uncomment блок + verify через staging.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:24:53 +03:00
lekss361
b3f32ae927
chore(infra): correct frontend healthcheck + tighten depends_on (re-apply #117) (#120)
Forward-fix после incident May 14 (см. vault events/Incident_Prod_Down_May14_2026).
PR #117 был полностью reverted PR #119 (Caddy depends_on frontend: service_healthy
+ failing wget --spider /  → Caddy не стартовал → vsё прод down ~25 мин).

Этот PR делает то же что планировал #117, но корректно:

### Frontend healthcheck — TCP probe через node (НЕ HTTP)

```yaml
test: ["CMD", "node", "-e",
  "require('net').createConnection({port:3000,host:'127.0.0.1'})
   .once('connect',function(){process.exit(0)})
   .once('error',function(){process.exit(1)})"]
start_period: 60s
retries: 6
interval: 15s
```

- НЕ зависит от HTTP status (200/404/308 — все OK если порт слушает)
- `node` гарантированно в alpine image (запускает server.js)
- `127.0.0.1` — никакой DNS ambiguity (busybox alpine localhost IPv6 vs IPv4)
- 60s + 6×15s = 150s window — с запасом на Next.js standalone cold start

### Caddy depends_on — backend healthy, frontend started (НЕ healthy)

```yaml
caddy:
  depends_on:
    backend: { condition: service_healthy }    # backend имеет /health endpoint
    frontend: { condition: service_started }   # safety net — НЕ блокирует Caddy
```

Lesson from incident: frontend health flaky (Next.js cold start variable).
Strict service_healthy для frontend → одна misconfig = full outage обоих доменов
(gendsgn.ru + obsidian.gendsgn.ru через тот же Caddy на shared network).

### backend/worker/beat redis: service_started → service_healthy

Redis уже имеет `redis-cli ping` healthcheck. Tightening безопасно — был
artifact pre-healthcheck Redis.

### Pin rosreestr2coord>=5.0.0 (lock уже 5.3.3)

v5 убрала delay параметр; код адаптирован (rate limit на worker уровне).
Защита от silent downgrade при `uv lock --upgrade`.

### Что НЕ повторено из #117

- Caddy depends_on frontend: service_healthy — заменено на service_started
- wget --spider / — заменено на node TCP probe

### Vault обновлён

- `events/Incident_Prod_Down_May14_2026.md` — постмортем + timeline + lessons
- Update Changelog в `domains/infra/infra-MOC.md` (если есть)

Closes incident. Re-applies #117 goals safely.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:24:47 +03:00
lekss361
6ea2dcd46c
Revert "chore(infra): healthcheck frontend + tighten depends_on; pin rosreestr2coord>=5 (#117)" (#119)
This reverts commit bade2f7772.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:02:58 +03:00
lekss361
bade2f7772
chore(infra): healthcheck frontend + tighten depends_on; pin rosreestr2coord>=5 (#117)
- Add wget healthcheck to frontend service (alpine-compatible) — Caddy
  больше не роутит трафик до того как Next.js готов.
- Caddy depends_on backend/frontend: service_started → service_healthy.
- backend/worker/beat redis dependency: service_started → service_healthy
  (Redis уже имеет redis-cli ping healthcheck).
- Pin rosreestr2coord>=5.0.0 (v5 убрала delay параметр; код уже адаптирован
  per CLAUDE.md); uv resolved 5.3.3.

Closes Day-1 quick wins #3+4 from code review audit (May 14).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 22:42:09 +03:00
lekss361
1c2f80a4b8
feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 FINAL) (#116)
* feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 #4 FINAL)

Замыкает Sprint 1.1 из #94 part 2 plan. После этого PR пользователь видит
свежие НСПД данные в UI (frontend integration — отдельный PR).

Backend (new app/services/site_finder/quarter_dump_lookup.py):
- `derive_quarter_cad(cad_num)` — 3/4/5-сегмент → quarter (3-segment)
- `get_quarter_dump_data(db, cad_num, parcel_wkt)` — main entrypoint:
  - Reads nspd_quarter_dumps row для derived quarter
  - Freshness threshold: 180 days
  - Missing/stale/harvest_error → trigger harvest_quarter.apply_async() fire-
    and-forget (lazy import против circular), return EMPTY_DUMP_RESULT
  - Fresh + parcel_wkt=None → metadata only (no spatial queries)
  - Fresh + geometry → 3 spatial queries via jsonb_array_elements + ST_Transform
    (3857→4326) + ST_Intersects / ST_DWithin
- 3 private helpers:
  - `_get_zoning` — point-in-polygon parcel centroid vs territorial_zones, LIMIT 1
  - `_get_zouit_overlaps` — все zouit_% layers пересекающиеся с parcel
  - `_get_engineering_nearby` — engineering_structures в 200m, sorted by distance
- `EMPTY_DUMP_RESULT` module-level constant — DRY для no-dump fallback (used
  in get_quarter_dump_data internal + analyze_parcel try/except wrap)

Backend (parcels.py):
- Import EMPTY_DUMP_RESULT + get_quarter_dump_data
- Call wrapped в try/except — если nspd_quarter_dumps недоступна (DB timeout
  / table missing) → EMPTY_DUMP_RESULT fallback вместо 500 (consistent с
  resilience pattern других optional fetches)
- Response gets 4 new fields:
  - nspd_zoning: dict | None (G1 ПЗЗ — zone_code, zone_name, source)
  - nspd_zouit_overlaps: list[dict] (G3 — overlaps в parcel, per ЗОУИТ group)
  - nspd_engineering_nearby: list[dict] (I3 — engineering structures в 200m)
  - nspd_dump: dict (freshness metadata — available, fetched_at_utc, stale,
    harvest_triggered, total_features)

Tests: 13 new в test_quarter_dump_lookup.py (mock-based, no real DB):
- derive_quarter_cad 5 edge cases (3seg, 4seg, 5seg, invalid, whitespace)
- get_quarter_dump no_row → harvest triggered
- stale (>180d) → harvest triggered, stale=True
- harvest_error row → retry harvest triggered
- parcel_wkt=None → metadata only (1 DB call)
- fresh + zoning extraction
- fresh + zouit_overlaps list
- fresh integration: все 4 keys present

47 pre-existing tests still pass.

Code review (code-reviewer pre-push): MINOR, 0 blocking. Applied 2 of 4:
-  #1: try/except wrap around get_quarter_dump_data в analyze_parcel
  (защита от DB unavailability) + DRY через EMPTY_DUMP_RESULT module const
-  #2: removed redundant nspd_zoning.fetched_at_utc (DRY — freshness в
  nspd_dump.fetched_at_utc)
- ⏭ Deferred (acceptable): #3 ad-hoc harvest_quarter retry cooldown для
  harvest_error rows (только при high traffic + persistent NSPD errors);
  #4 raw_props в response — tech debt, убрать вместе с frontend PR

Performance note: 3 spatial queries per analyze adds ~10-50ms on typical
~100-feature quarter. Mitigation if quarters grow dense: materialized
per-layer sub-table (отдельная DB issue).

Closes Sprint 1.1 part of #94. Frontend rendering этих 4 полей — отдельный
PR (next: #112 / #115 / #114).

* fix(site-finder): address PR #116 auto-review M1-M5

M1 (mutation risk): replace EMPTY_DUMP_RESULT direct refs with
_make_empty_result() factory. dict(...) shallow copy left nested
nspd_dump shared by reference across concurrent requests — single
mutation pollutes module sentinel for all subsequent calls. Now
каждый caller gets independent dict.

M2 (O(N) spatial scan): SELECT extended denormalized counts
(territorial_zones_count, zouit_count, engineering_count). Each
spatial helper accepts layer_counts and early-returns when count=0
— skips heavy jsonb_array_elements + ST_Transform/ST_Intersects
scan entirely. Critical для quarters с 2000+ features.

M4 (documentation): _trigger_harvest docstring describes known
burst/no-dedup limitation + TODO Redis SETNX (отдельный PR).

M5 (test fragility): _make_db_mock_with_spatial docstring describes
positional-call contract — db.execute order (0=dump, 1=zoning, 2=zouit,
3=engineering) и зависимость от count-values.

+4 new tests (17 total, all pass):
- test_make_empty_result_returns_independent_copies (mutation safety)
- test_make_empty_result_overrides
- test_early_exit_all_counts_zero_no_spatial_queries
- test_early_exit_partial_counts

Per auto-review on 3068a9c.

* fix(site-finder): rename _make_empty_result → make_empty_result (public) per PR #116 review

M1 residual fix: parcels.py exception path использовал EMPTY_DUMP_RESULT
singleton ref вместо factory. Сейчас readonly access, но нарушает
documented invariant модуля.

Rename `_make_empty_result` → `make_empty_result` (public API), import в
parcels.py, использовать в try/except fallback. Каждый request получает
независимый dict — никаких shared references.

M4 (Redis SETNX dedup) + M5 (test fragility) — deferred per review,
documented в code/issue. Acceptable trade-offs:
- M4: UPSERT idempotency делает данные safe; burst-duplicate task'и тратят
  WAF traffic впустую но не повреждают данные.
- M5: docstring contractually describes positional-call order.

17/17 tests pass. ruff/format clean.

Per auto-review on aef8308.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-13 09:14:19 +03:00
lekss361
f3b0ce5687
feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4 part A) (#111)
* feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4/4 part A)

Sprint 1.1 item #3 из #94 part 2. Замыкает harvest cycle: Celery task
оборачивает NSPDClient.search_by_quarter (PR #109) + UPSERT в
nspd_quarter_dumps (PR #110). После — analyze_parcel будет читать dumps
вместо on-demand NSPD HTTP в request-цикле (Sprint 1.1 #4, следующий PR).

Backend:
- New `backend/app/workers/tasks/nspd_sync.py`:
  - `harvest_quarter(quarter_cad, region_code=66, include_zouit=True,
    include_risks=False)` — single-quarter harvest. autoretry_for=
    (NspdLiteWafError,) с retry_backoff=True, max_retries=3,
    soft_time_limit=120s. Generic exception → upsert error row с
    harvest_error=str(exc), counts=0 (НЕ re-raise).
  - `harvest_stale_quarters(region_code, max_age_days=90, batch_size=50)` —
    beat fanout: SELECT cad_quarters_geom EXCEPT fresh dumps → apply_async per cad.
  - UPSERT использует ST_Multi(ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON,3857),4326))
    для quarter_geom (schema MultiPolygon strict per database-expert contract).
- `celery_app.py`:
  - Add `nspd_sync` в `include`
  - Beat entry `nspd-harvest-stale-quarters` Mon 04:00 МСК, batch_size=50,
    max_age_days=90
  - `worker_ready` sanity check `SELECT 1 FROM nspd_quarter_dumps LIMIT 0`
    (non-fatal critical log per Bug_Worker_Ready_EarlyReturn_Fixed pattern)
- `admin_scrape.py`:
  - `HarvestQuarterRequest` Pydantic с pattern `^\d+:\d+:\d+$` (quarter only)
  - `POST /api/v1/admin/scrape/nspd/harvest-quarter` manual trigger,
    requires X-Admin-Token, returns task_id.

Tests (`test_nspd_sync.py`): 14 mock-based (no real NSPD/DB):
- happy path: search_by_quarter mock → verify SQL params/structure
- empty quarter → row with quarter_geom=NULL, total=0, no harvest_error
- WAF error → autoretry kicks in (raise re-propagated)
- Generic error → upsert error row
- Fanout: 50 stale → 50 apply_async calls
- Layer names preservation (all 5 zouit + 11 risks if enabled)
- Geometry SRID handling

Code review (code-reviewer pre-push): APPROVE, 0 blocking, 5 minor deferred:
- inline datetime import in error branch (style)
- layers_fetched array literal via string concat (current layer names safe)
- harvest_stale_quarters error rows always re-tried (intended, undocumented)
- test call_args.kwargs fragile (works for current callsite)
- beat collision with refresh-ekb-districts-medians every ~2 years (negligible)

Part of #94. Next: Sprint 1.1 #4 — analyze_parcel reads dump cache.

* fix(nspd-sync): address PR #111 auto-review M1-M5+M7 findings

M1 (runtime bug): remove WHERE region_code from outer cad_quarters_geom SELECT
in harvest_stale_quarters — column does not exist in prod schema
(cad_number, geom, raw_props, fetched_at, source). Would crash on first beat
tick. Inner nspd_quarter_dumps subquery keeps region_code filter unchanged.

M2 (correctness): _upsert_dump теперь делает db.rollback() в except перед
db.close() — prevents PendingRollbackError на следующем session use, если
PostGIS отбракует geometry или JSON cast failure.

M3 (low): worker_ready breadcrumb INSERT тоже теперь имеет rollback —
consistency с остальными session usages в celery_app.py.

M4 (style): layers_fetched — заменил manual string concat
'{' + ','.join(...) + '}' на native Python list. psycopg v3 сам сериализует
list → text[]. Защита от future layer names с запятыми/фигурными скобками.

M5 (style): import datetime as _dt поднят из else branch в top-level imports.

M7 (test cleanup): удалил dead __enter__/__exit__ mock setup в
test_harvest_stale_quarters_fanout / _empty. harvest_stale_quarters не
использует session как context manager — moc'и no-op.

14/14 tests pass. ruff/format clean.

Per auto-review on 9b2a289.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 18:45:57 +03:00
lekss361
9ee0b07003
feat(scrapers): search_by_quarter orchestrator + QuarterDump (#94 pt.2/4) (#109)
Sprint 1.1 item #1 из плана #94 part 2. Foundation для PKK harvest pipeline —
1 vacuum (search) + N layer fetches → comprehensive snapshot всех NSPD данных
в пределах квартала. Базис для G1 #28 ПЗЗ, G3 #30 ЗОУИТ, P2 #46 neighbors,
E1 #51 parcels backfill, #96 ЕГРН помещения.

Backend (nspd_client.py):
- New QuarterDump frozen dataclass (slots=True): quarter + per-layer feature
  lists (parcels/buildings/territorial_zones/red_lines/engineering + zouit
  dict + risks dict) + bbox_3857 + layers_fetched (immutable tuple) +
  fetched_at_utc + total_features property.
- New NSPDClient.search_by_quarter(quarter_cad, include_zouit=True,
  include_risks=False): search → bbox → bulk fetch per layer phase.
  Cost 6/11/22 requests.
- New _geojson_bbox_3857() module-level helper — recursive coord walker.
- Class constants QUARTER_CORE_LAYERS / QUARTER_ZOUIT_LAYERS /
  QUARTER_RISK_LAYERS.

Empty-quarter (NSPD не нашёл cad): quarter=None, bbox=None, all lists empty,
zouit/risks dicts populated с пустыми lists (структурно стабильно),
layers_fetched=('search',).

Tests: +12 tests (31 total, no network).

Code review (code-reviewer pre-push): MINOR, fixed 3 of 5:
- datetime import → module-level
- layers_fetched → tuple[str, ...] (immutable in frozen dataclass)
- docstring clarified empty-quarter semantics
Bonus: ruff UP038 isinstance tuple → union syntax.

Part of #94. Sprint 1.1: 4 PRs total. Next: migration → Celery → integration.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 18:11:30 +03:00
lekss361
bc092b58f1
feat(scrapers): NSPD client foundation (#94) — search + WMS + layers (#98)
* feat(scrapers): NSPD client foundation (#94) — search_by_cad + WMS + layers

Foundation для G1 #28 ПЗЗ, G2 #29 ВРИ, G3 #30 ЗОУИТ, E1 #51 cad_parcels,
I3 #44 engineering, и поддержки on-demand #93.

Backend (`app/services/scrapers/nspd_client.py`):
- `NSPDClient` с 4 core методами:
  - `search_by_cad(cad, thematic_id)` — REST /api/geoportal/v2/search
    возвращает GeoJSON + ВРИ + land_category + cost_value за 1 запрос
  - `get_feature_info(layer_id, lon, lat, buffer_m=100)` — WMS GetFeatureInfo
    на точке: какие feature'ы layer'а покрывают (lon,lat)
  - `get_features_in_bbox(layer_id, bbox_3857)` — bulk через WMS workaround
    (WFS GetCapabilities → 404, поэтому большой bbox + центральная I/J точка)
  - `list_layers(theme_id)` — каталог слоёв в теме (PKK=1, ARN=665)
- Typed responses: NSPDFeature, NSPDSearchResult, NSPDLayer (frozen dataclasses)
- LAYERS catalog: 32 layer-id с семантическими именами (territorial_zones=875838,
  zouit_engineering=37578, risk_flooding=872205, etc) — TIER 1-6 per #94
- Coordinate helpers: lonlat_to_3857(), bbox_around_point_m()
- Reuse: HEADERS + SSL ctx + fetch_geoportal из existing nspd_lite (WAF-compatible
  urllib trick). Rate limit через rate_ms.
- WAF/Rate-limit: raises NspdLiteWafError на 403/429 — caller backoff.

Tests (`backend/tests/test_nspd_client.py`): 14 unit tests, no network.
- LAYERS catalog sanity (territorial_zones=875838 закрывает G1)
- Coordinate transforms (zero, ЕКБ center, bbox)
- NSPDFeature.from_raw parsing (full / missing fields)
- NSPDSearchResult helpers (empty/first)
- _walk_layer_tree (flat / nested / empty)
- search_by_cad через monkeypatch fetch_geoportal

Также включён micro-fix из PR #95 review (cosmetic):
- parcels.py inline comment "до 25s" → "до 15s" (matched _INLINE_FETCH_WAIT_S)

Scope NOT в этом PR (отдельные follow-up issues, sequential rule):
- Schema migrations (cad_parcels_geom +columns, nspd_territorial_zones,
  nspd_zouit, nspd_red_lines, nspd_risk_zones tables) → отдельный PR
- Celery tasks (sync_territorial_zones_bbox, sync_zouit_*, sync_risk_*) →
  отдельный PR использует client
- Admin endpoints (trigger / status) → отдельный PR
- Замена on-demand fetch на nspd_client → PR after schema ready

Closes part of #94 (foundation only — sub-issues for schema/tasks).

* fix(scrapers): address PR #98 auto-review minor feedback

Backend (nspd_client.py):
1. (line 100+) bbox_around_point_m signature split на multi-line.
2. (lazy imports) import json, time → module-level.
3. (duplicate SSL ctx) Reuse _SSL_CTX из nspd_lite через explicit import.
4. (shape defense) list_layers чекает {"data": [...]} wrapper + non-dict/list garbage → warning + empty.
5. (cad_num docstring) search_by_cad документирует формат + ссылку на validate_cad_format.

Tests: +5 mock тестов (19/19 passed):
- get_feature_info URL builder + parsing
- get_features_in_bbox bbox propagation
- list_layers_walks_tree_response
- list_layers_handles_data_wrapper
- list_layers_handles_garbage_response

Backend (parcels.py): docstring '~25с' → '~15с'.

Per auto-review on be14388.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 09:47:45 +03:00
lekss361
49fac806fc
feat(site-finder): auto-fetch cadastre geometry on-demand (#93) (#95)
* feat(site-finder): auto-fetch cadastre geometry on-demand (#93)

Когда пользователь вводит cad-номер которого нет в БД (cad_quarters_geom /
cad_buildings / cad_parcels_geom), вместо 404 «Загрузи через NSPD geo» (dead-
end для non-admin) — теперь backend автоматически инициирует NSPD fetch,
frontend показывает loading state с polling /fetch-status каждые 2с.

Backend:
- Новый модуль app/services/site_finder/cadastre_fetch.py с helpers:
  find_or_enqueue_fetch (atomic check + enqueue с дедупликацией по cad),
  fetch_status (smart polling endpoint — отличает not_in_nspd от failed),
  detect_thematic_id (3-сегм quarter / 4-сегм parcel / 5-сегм building),
  validate_cad_format.
- Reuse: enqueue_geo_job + process_nspd_geo_job (workers/tasks/nspd_geo).
  source_kind='auto_on_demand' отличает от bulk; rate_ms=200, priority=9.
- POST /parcels/{cad}/analyze graceful fallback: inline await до 25s
  (fast path), затем 202 + job_id + eta_seconds для polling, либо
  400/404/503 в зависимости от статуса (с Retry-After 60s на 503).
- GET /parcels/{cad}/fetch-status новый endpoint для polling.

Frontend:
- useSiteAnalysis расширен: fetchingState + cancel(). POST analyze + при 202
  polling каждые 2с (max 60 итераций = 2 мин cap). status=ready → re-trigger
  analyze; not_in_nspd/failed/invalid_format → typed errors.
- apiFetchWithStatus<T> + HTTPError в lib/api.ts — status-aware variant
  для 200 vs 202.
- FetchingState.tsx: animated spinner, progress bar (linear до etaSeconds,
  asymptote после), elapsed counter, cancel button. Светло-голубая
  scheme отличается от обычного pending skeleton.
- site-finder/page.tsx: FetchingState когда fetchingState активен; обычный
  pending skeleton — только при первичном analyze без 202.

Edge cases (per #93 acceptance):
- cad валидный в НСПД, fetch <25s → inline 200 OK
- cad валидный, fetch >25s → 202 + frontend polls → ready → analyze
- cad валидный, не в НСПД → 404 с понятным сообщением + формат hint
- cad invalid format → 400 + формат hint
- NSPD rate-limited / failed → 503 + Retry-After 60s
- Параллельные запросы на тот же cad → один job, оба клиента poll'ят (дедуп
  через find_active_on_demand_job).

Closes #93.

* fix(site-finder): address PR #95 auto-review minor feedback

Backend (cadastre_fetch.py):
1. (race condition) Advisory lock `pg_try_advisory_xact_lock(hashtext(cad_num))`
   обёрнут вокруг шагов "check active job → enqueue" в find_or_enqueue_fetch.
   Lock transaction-scoped, released at COMMIT. Параллельные запросы на тот
   же cad: первый получает lock и enqueue; второй lock=false → re-check
   active job (уже виден после первого COMMIT) → возвращает тот же job_id.
   Docstring обновлён, упоминание SELECT FOR UPDATE удалено.

Backend (parcels.py):
3. (threadpool exhaustion) _INLINE_FETCH_WAIT_S снижен 25 → 15s с подробным
   комментарием: tradeoff про default Starlette anyio threadpool (40 slots)
   и concurrent burst saturation. 15s баланс: НСПД avg 5-15s для quarter,
   ~70% fast path; остальные 30% получают 202 + polling без блока.

Data (87_on_demand_indexes.sql):
2. (missing index) New migration:
   - `nspd_geo_targets_cad_num_idx ON nspd_geo_targets(cad_num)` — для
     find_active/recent_completed_job (existing UNIQUE composite не покрывает
     WHERE cad_num=:c).
   - `nspd_geo_jobs_source_status_idx ON nspd_geo_jobs(source_kind, status)`
     composite для filter auto_on_demand + queued/running.
   Idempotent (CREATE INDEX IF NOT EXISTS), не блокер при текущем размере,
   критично при росте on-demand traffic.

Frontend (useSiteAnalysis.ts):
4. (UI flicker) setFetchingState(null) перемещён ПОСЛЕ `await second
   apiFetch`. Иначе между clear и resolve есть момент когда mutation
   isPending=true + fetchingState=null → пустой экран ~1 RTT.

NOT addressed (rebuttal):
5. (Tailwind convention) — проверил: в проекте нет ни globals.css, ни
   @tailwind directives. ВСЕ существующие site-finder components используют
   inline styles (ConfidenceBadge, GeotechRiskBlock, ScoreBreakdownPanel etc).
   Tailwind в deps но не wired up. Keep inline styles для consistency.
6. (animate-spin) — требует Tailwind globals (см. #5). `<style jsx>`
   keyframes — built-in Next.js, работает.

Per auto-review on 2252236.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 09:02:17 +03:00
lekss361
f4a865060f
feat(site-finder): D4 pipeline 24mo — future competition (#36) (#90)
* feat(site-finder): D4 pipeline 24mo — future competition (#36)

Backend (parcels.py):
- Запрос к domrf_kn_objects в радиусе 5км с ready_dt BETWEEN NOW() AND NOW()+24mo.
- _aggregate_pipeline() — сводка: objects_count, flats_total, by_class (эконом/
  комфорт/бизнес/...), by_quarter (хронологически, для UI bar), severity
  (low <500 / medium <3000 / high) per spec, top_objects (десятка по flat_count desc).
- Поле analyze.pipeline_24mo. Backward-compat — optional.

Frontend:
- Pipeline24moBlock.tsx — severity badge + 3 summary numbers (объектов, квартир,
  горизонт/радиус), by-class chips, гистограмма bar по кварталам сдачи
  (нормирована на max), разворачиваемый top-N список с классом + датой сдачи.
- Добавлен в MarketTab выше "Market trend".
- TS типы: Pipeline24mo, PipelineObject, PipelineQuarterSlot.

Closes #36. Relates to #19 (Конкурентный 360 — закрыт ранее в #36 scope).

* fix(site-finder): address PR #90 auto-review feedback

Must-fix (3):
1. distance_m falsy guard: `if obj.get("distance_m") is not None` вместо
   `if obj.get("distance_m")` — centroid-on-building даёт 0.0 (falsy float),
   raw Decimal иначе упал бы в JSON serialization.
2. SQL plan note добавлен про seq scan ~3000 строк OK; при росте — нужен
   GIST/index на (latitude, longitude) — отдельный issue для database-expert
   (будет создан separately).
3. obj_class NULL bug помечен в docstring _aggregate_pipeline с reference на
   fixes/Bug_Kn_API_Obj_Class_Always_Null_OPEN. D6/#38 — fix плановый.

Cleanup (3 из 5):
4. CLASS_LABEL.null dead key убран — JSON null приходит as absent key, не
   "null" string.
6. Magic numbers вынесены: PIPELINE_RADIUS_M=5000, PIPELINE_HORIZON_MONTHS=24,
   PIPELINE_SEVERITY_MEDIUM_THRESHOLD=500, PIPELINE_SEVERITY_HIGH_THRESHOLD=3000,
   PIPELINE_TOP_OBJECTS_LIMIT=10. SQL query теперь через f-string подставляет
   их (защищённое от injection — это int литералы).
8. obj.ready_dt formatting через fmtMonth() с new Date + toLocaleDateString —
   robust к datetime suffix vs date-only, fallback к substring(0,7) при NaN.

Не сделано (defer):
5. Async 3 HTTP calls (pre-existing pattern, нужен ThreadPoolExecutor refactor
   отдельным PR — затрагивает weather/air_quality fetch architecture).
7. ST_GeomFromText дважды — CSE справляется на этом масштабе.

Per auto-review on ade511b.

* fix(site-finder): address PR #90 auto-review minor feedback

1. TS PipelineObject.distance_m — `number | null` для отражения defensive
   Python guard (`if obj.get("distance_m") is not None`). Comment объясняет
   почему.
2. Pipeline SQL: `text(f"...")` → `text("...")` + parameters. radius_m и
   horizon_months через `:param` placeholders + `cast(:horizon_months || ' months'
   AS interval)`. Consistency с остальными SQL в файле, plus защита от
   accidental injection при будущих изменениях.
3. top_objects: explicit field selection вместо `dict(r) for r in rows`.
   Раньше leak'ило все колонки из CTE `SELECT *` (latitude/longitude/
   snapshot_date/region_cd/dev_id) в API response. Теперь только nominated
   fields: obj_id, comm_name, dev_name, obj_class, flat_count, ready_dt,
   distance_m. Schema clean.

Per auto-review on 4e431bf.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 08:34:02 +03:00
lekss361
1f8fd77825
feat(site-finder): X1 score breakdown + verbal explain (#47) (#92)
Backend (parcels.py):
- POI scoring loop teper строит score_breakdown_detailed: per-factor list с
  verbal explain (через _verbal_for_poi helper) и группировкой
  (_POI_GROUP: Социалка / Торговля / Парки / Транспорт / Шум/трамвай / Локация).
- center_bonus добавлен как synthetic factor группы "Локация" с weight=1.0
  (decay не применяется — bonus IS the value).
- factor key включает enumerate idx — prevents React key collision когда
  два POI одной категории совпадают по округлённому расстоянию.
- Skip факторов с |contribution| < 0.01 (POI > 1км) — UI шуму не нужен.
- abs_total fallback на 1.0 — защита от division-by-zero для empty factors.
- Top-3 positives/negatives: explicit ascending sort для негативов
  ("most-negative first" очевидно из кода).
- score_by_group: stacked-bar данные с count + contribution_pct.
- group_totals type: dict[str, dict[str, float | int]] (count это int).

Frontend:
- Новый ScoreBreakdownPanel.tsx: stacked-bar по группам с tooltip + legend,
  топ-3 плюса (▲ зелёный) / топ-3 минуса (▼ красный) с verbal, отдельная
  строка "Снижают балл — Шум/трамвай: ..." для negative groups, разворачиваемая
  таблица всех факторов (sticky thead, scrollable).
- Интегрирован в OverviewTab под секцией "Балл".
- TS типы: FactorContribution, ScoreGroupTotal.

Closes #47.

NB: branch создан заново из-за rebase mess (см. PR #87 comments). Логически
эквивалентно но history clean.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 08:17:00 +03:00
lekss361
09611dc690
feat(site-finder): P1 geometry suitability score (#45) (#89)
* feat(site-finder): P1 geometry suitability score (#45)

Backend (parcels.py):
- _polygon_suitability() — Shapely metrics on parcel WGS84 polygon:
  area_ha/m2 (projected via cos(lat) к UTM-like meters), perimeter_m,
  aspect_ratio (long/short side of MABR), convex_hull_ratio (area / hull area),
  min_inscribed_rect_dim_m.
- Composite suitability score 0..1:
  base = area_subscore (0 при <0.2 ha, linear → 1 при ≥0.5 ha)
  −0.3 если aspect_ratio > 5 (вытянутый)
  −0.3 если convex_hull_ratio < 0.65 (изрезанный)
  −0.5 если min_inscribed_rect_dim_m < 30 (узкий)
- Label: микро / подходящий / сложная форма / слабо подходит
- Recommendation: "для МКД 16+ нужно >0.3 ha + минимум 40м"
- В analyze response → geometry_suitability.

Frontend:
- Новый GeometrySuitabilityBlock.tsx с color-coded badge + метрики grid +
  penalties + recommendation
- Добавлен в LandTab (выше "Зонирование (ПЗЗ)")
- TS типы расширены: GeometrySuitability

Closes #45.

* fix(site-finder): address PR #89 auto-review minor feedback

Backend (parcels.py):
1. Фактическая ошибка фикс: recommendation теперь различает "строительный
   минимум 30м" (physical penalty trigger) и "комфорт МКД 12-16эт 40м"
   (recommendation level). Пользователю чётко видны 2 порога.
2. Lazy imports → module-level: `from shapely import wkt as _shp_wkt`,
   `from shapely.geometry import Polygon`.
3. MABR inner exception теперь логирует: `logger.debug("MABR computation
   failed, falling back to sqrt(area): %s", mabr_err)`.
4. Magic numbers вынесены в константы: _GEOM_MIN_AREA_HA, _GEOM_AREA_SCORE_FULL_HA,
   _GEOM_ASPECT_PENALTY_THRESHOLD, _GEOM_CONVEX_PENALTY_THRESHOLD,
   _GEOM_MIN_WIDTH_PHYSICAL_M, _GEOM_MIN_WIDTH_COMFORT_M, _GEOM_LABEL_MICRO_HA,
   _GEOM_LABEL_GOOD, _GEOM_LABEL_MEDIUM, плюс penalty константы.
5. Label "микро" теперь комбинируется с penalties: "микро, узкий" — UI видит
   обе проблемы. Pure "микро" остаётся когда нет penalty.

Frontend (GeometrySuitabilityBlock.tsx + types):
6. Заменил emoji ⚠ (U+26A0) на текстовый <strong>"Проблемы формы:"</strong> —
   стабильнее в WeasyPrint PDF-экспорте и cross-platform.
- TS типы: GeometrySuitabilityBaseLabel экспортирован, label расширен до
  `string` для допуска combo-labels; helper colorForLabel() парсит base часть.

Per auto-review on f4e7491.

* fix(site-finder): P1 #45 — extract 0.3 ha to _GEOM_AREA_COMFORT_HA

Auto-review нашёл: recommendation упоминала "от 0.3 га" но в constants block
не было 0.3 — magic literal в строке. Тот же класс проблемы что 30/40м inconsistency
из прошлого review, только в другом поле.

Fix: новая константа `_GEOM_AREA_COMFORT_HA = 0.3` с комментарием
"рекомендуемая комфортная площадь МКД (recommendation)". Размещена между
_GEOM_MIN_AREA_HA (физический минимум) и _GEOM_AREA_SCORE_FULL_HA (premium) —
третий semantic threshold. Recommendation теперь f-string использует константу.

Per auto-review on ae5e8de.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 08:00:12 +03:00
lekss361
843314c040
feat(site-finder): P2 cad_buildings соседи + overlap check (#46) (#91)
* feat(site-finder): P2 cad_buildings соседи + overlap check (#46)

Backend (parcels.py):
- _parse_floors() helper для TEXT column (cad_buildings.floors хранится как
  строка, могут быть диапазоны "5-7"). Возвращает верхнюю границу.
- _neighbors_summary(db, geom_wkt, our_cad) — query соседей в 100м (GIST):
  cad_num, building_name, floors, year_built, cost_value, area, address, distance.
  Aggregate: avg_floors_100m, max_floors_100m, median_cost_per_m2_100m,
  count_buildings_100m. Outliers cost/m² фильтруются (1k < x < 500k).
- Overlap check: ST_Intersects + ST_Area(ST_Intersection) > 50 m² (transformed
  to UTM 32641 для метров). Если есть → has_existing_buildings: true +
  overlap_buildings list.
- В response → neighbors_summary.

Frontend:
- Новый NeighborsBlock.tsx: hard red warn block для overlap (с building names +
  overlap_m2 + "Инвестиции невозможны без сноса"); summary metrics (avg/max
  floors, median price); toggle "Показать N ближайших" → таблица.
- Border меняется на красный при has_existing_buildings — visual cue.
- Добавлен в LandTab выше "Зонирование (ПЗЗ)".
- TS типы: NeighborBuilding, OverlapBuilding, NeighborsSummary.

Closes #46. Closes #21 (cad_buildings в Site Finder фильтрах).

* fix(site-finder): address PR #91 auto-review minor feedback

Backend (parcels.py):
1. (medium) Aggregation loop _neighbors_summary теперь обёрнут в try/except
   (ValueError, TypeError) с fallback к data_available:False + log warning.
   Защищает от non-numeric cost_value/area придёт в строке (e.g. "N/A") —
   ранее весь endpoint падал 500.
2. Magic numbers вынесены: _COST_PER_M2_MIN=1000, _COST_PER_M2_MAX=500_000.
3. _parse_floors docstring + inline note про malformed parts ("5а-7" filter,
   multi-range "1-2-3" max acceptable degradation).

Frontend (NeighborsBlock.tsx):
5. Русский plural fix: pluralBuildings(n) helper — 1 здание, 2-4 здания,
   5+/11-14 зданий. Раньше "3 зданий" — теперь "3 здания".

Не сделано (defer):
4. ST_Area для overlap query — практически 0-5 buildings в ЕКБ, GIST scan OK.
6. Discriminated union для NeighborsSummary — refactor а не bug.
7. Vault entry для P2 — добавится batch'ем после merge всех текущих PR.

Per auto-review on 60d53bb.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 01:30:53 +03:00
lekss361
3777a77a42
feat(site-finder): X2 confidence indicator + caveats (#48) (#88)
Backend (parcels.py):
- _compute_confidence() composite score 0..1 from 7 subscores: poi_freshness,
  geom_source (parcel vs quarter), district, market_trend (rosreestr_deals depth),
  competitors, environment (noise/air/weather availability), zoning (placeholder
  до G1).
- confidence_label: high (>0.75) / medium (0.4-0.75) / low (<0.4)
- confidence_caveats: list of конкретных проблем для UI
- confidence_breakdown: per-subscore 0..1 для прозрачности

Это stub-версия (полная — после G1/G2/D1/D2). Использует только текущие сигналы.

Frontend:
- Новый ConfidenceBadge.tsx — color-coded (green/yellow/red) badge с %
- Caveats для low — показываются сразу; для medium/high — под toggle
- Toggle "Подробнее" → breakdown per-subscore + полный список caveats
- Размещён в начале OverviewTab (выше "Район")
- TS типы расширены: confidence, confidence_label, confidence_breakdown, confidence_caveats

Closes #48.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 01:04:25 +03:00
lekss361
2cbcea9e73 fix(zoning): NSPD blocks bot access — fallback to deep-links (PKK6 API closed 2024) 2026-05-11 22:56:50 +03:00
lekss361
49eadeb9ce fix(pzz-sync): follow_redirects=True (PKK6 returns 301) 2026-05-11 22:50:46 +03:00