Commit graph

206 commits

Author SHA1 Message Date
lekss361
5da66bbc93
feat(cadastre): bulk_harvest worker + grid-walker + admin API (#168 PR3/5) (#171)
* feat(cadastre): bulk_harvest_quarter Celery task + grid-walker + saga state (#168 PR3/5)

Add bulk cadastre harvest pipeline:
- services/cadastre/bulk_harvest.py: async harvest_quarter() orchestrator (4 phases)
  + 7 upsert helpers (parcels/buildings/constructions/oncs/enks/zouit/quarter_stats)
  using CAST(:x AS jsonb) pattern, begin_nested() SAVEPOINT per grid-walk upsert
- services/cadastre/grid_geometry.py: quarter_bbox_3857 (PostGIS ST_Extent)
  + generate_grid_click_points (15x15 = 225 sub-bbox grid)
- workers/tasks/scrape_cadastre.py: bulk_harvest_quarter_task (acks_late=True,
  dont_autoretry_for NspdBulkWafError), enqueue_cadastre_harvest, cleanup_zombies
- api/v1/admin_cadastre.py: 5 endpoints behind AdminTokenAuth — create/list/get/cancel/resume
- Tests: 13 service unit + 8 API tests

* fixup(cadastre): drop importorskip (PR2 merged) — imports now top-level (#168 PR3)

* fixup(cadastre): tile_size unified param + register slow marker (#168 PR3)

Blocker #1: rename tile_width/tile_height → tile_size in generate_grid_click_points.
Callers in bulk_harvest.py and test_cadastre_bulk.py used tile_size; def had
tile_width+tile_height → TypeError at runtime.

Blocker #2 (10 failures in test_admin_cadastre.py) was side-effect of #1:
TypeError at import chain (app.main → admin_cadastre → bulk_harvest)
broke FastAPI app load → dependency_overrides AttributeError. Now resolved.

Also register `slow` pytest marker in pyproject.toml to suppress
PytestUnknownMarkWarning.

* fixup(cadastre): use importlib.util.find_spec instead of import (#168 PR3)

Root cause Blocker #2: `import app.workers.tasks.scrape_cadastre` at module
top-level REBOUND `app` from FastAPI instance (line 14) to the Python package.
All subsequent `app.dependency_overrides[get_db] = ...` failed with
AttributeError because `app` was now the namespace module.

Fix: probe module availability with importlib.util.find_spec — does not
import or rebind names. FastAPI `app` instance stays intact.

All 28 cadastre tests pass locally.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 13:31:32 +03:00
lekss361
59f222eaf3
feat(db): bulk cadastre schema — 7 tables + cadastre_jobs (#168 PR1/5) (#169)
* feat(db): schema migration for bulk cadastre ingest layers (#168)

Add 7 cadastre layer tables + cadastre_jobs saga-state table.

Tables:
- cad_parcels (36368 ЗУ ЕГРН) replaces cad_parcels_geom
- cad_buildings v2 wide schema (replaces existing, backed up as cad_buildings_old_apr26)
- cad_constructions (36383 Сооружения + ЕГРН, flag is_egrn)
- cad_oncs (36384), cad_enk (39663), cad_zouit (4 categoryIds), cad_quarter_stats
- cadastre_jobs (resumable saga state, mirrors geo_jobs pattern)

GIST + BTREE indexes. Migration of cad_parcels_geom data via INSERT...SELECT.
Restore v_complex_full + v_complex_buildings views after table rename.

* fixup(db): address PR #169 bot review — 2 blockers + 3 lessers

Blocker #1: cad_parcels_geom DROP → back-compat VIEW
  Old DROP TABLE would break site-finder analyze fallback (per 83_*.sql COMMENT).
  Now: drop old BASE TABLE (guarded by table_type), create VIEW over cad_parcels
  with identical 4-column signature. Callers continue to work; PR2/5 will switch.

Blocker #2: K.3 redundant RENAME CONSTRAINT removed
  ALTER INDEX RENAME on PK index already renames the backing constraint in PG16.
  Second ALTER TABLE RENAME CONSTRAINT would throw 'does not exist' and abort tx.

Lesser #1: cadastre_jobs.status CHECK constraint
  Inline CHECK (status IN ('queued','running','done','failed','zombie','cancelled','paused'))
  per partial-index intent and admin UI plan.

Lesser #2: COMMENT ON TABLE for all 8 tables + view (keeps convention from 63, 83).

Lesser #3: set_updated_at() triggers on 7 tables with updated_at column.
  Reuses existing function (verified present). DO $$ guards on pg_trigger
  for idempotent re-apply. cadastre_jobs excluded (uses heartbeat_at instead).

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 13:05:25 +03:00
lekss361
b7889d113b
feat(cadastre): NSPDBulkClient + schemas for bulk ingest (#168 PR2/5) (#170)
Add async NSPD client (nspd_bulk_client.py) with 3 new methods:
- search_by_quarter: REST /api/geoportal/v2/search/geoportal — snapshot 20 per cat + meta totals
- wms_feature_info: WMS GetFeatureInfo для grid-walk discovery
- list_objects_in_building: tab-group-data (Q3 deferred — код ready)

Rate limit: asyncio.Semaphore(3) + 0.05s jitter.
429 → exponential backoff 2→4→8s, max 3 retries.
403 (WAF) → raise immediately, no retry.

Pydantic schemas (nspd_bulk.py): NSPDBulkFeature, QuarterSnapshot, ObjectsListing
with extra='allow' to preserve unknown fields → raw_props JSONB downstream.

15 unit tests + 4 e2e (real NSPD, skipped in CI by default).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 12:50:41 +03:00
lekss361
b4327635fa
fix(site-finder): visible by-quarter pipeline bars (#167)
Bars rendered as ~1-15px because outer flex container had
align-items:flex-end which made each column shrink to label
height (~15.8px), so bar percentage heights resolved against
that instead of the intended 80px container.

Switched to align-items:stretch + fixed-pixel bar heights
anchored to a 60px area (reserves ~20px for label row). Each
column now uses justify-content:flex-end so bars + labels sit
at the bottom baseline.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 10:35:51 +03:00
lekss361
faa99abb94
feat(site-finder): per-room bucket breakdown в VelocityBlock (#166)
VelocityBucketStat + optional by_room_bucket fields.
'По комнатности' aggregate table + per-competitor inline mini-row.
Backward compat via optional field presence guards.

Closes #161 (frontend part)

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 10:12:42 +03:00
lekss361
d7fbaa0528
feat(velocity): per-room-bucket breakdown in VelocityResult (#163)
* fix(velocity): :window_interval::interval cast syntax — same bug as PR #152

SQLAlchemy text() parser confuses ':window_interval' (named param) followed
by '::interval' (cast operator) — exactly как было в PR #152 с :weights::jsonb.

Result: psycopg видит literal ':window_interval' в SQL → syntax error →
exception caught by velocity catch → return None → UI shows null.

## Fix

':window_interval::interval' → 'CAST(:window_interval AS interval)' (2 мест:
sales_rows query + _get_ekb_median percentile).

Pre-push code-reviewer должен был catch — добавим в feedback что для каждого
text() SQL grep ':[a-z]*::' before push.

Refs: PR #158 deploy verify, e2e velocity = null root cause

* feat(velocity): add per-room-bucket breakdown to VelocityResult

Add third SQL query (bucket_rows) aggregating deals_total_count by
room_bucket ('студия','1','2','3','4+') across mapped competitors.

New fields:
- VelocityResult.by_room_bucket: aggregate {units, sqm, complexes_count}
- sample_competitors[].by_room_bucket: {bucket: units}

Bucket query wrapped в SAVEPOINT — failure degrades gracefully к empty
dict без abort outer tx.

CAST(:window_interval AS interval) pattern per PR #160 lesson.

Test coverage: 3 new tests (aggregation, empty fallback, sample entries).

Closes part of #161

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 10:09:03 +03:00
lekss361
30d78c5030
fix(pipeline): enrich obj_class via objective_lots COALESCE fallback (#165)
domrf_kn_objects.obj_class всегда NULL (Bug_Kn_API_Obj_Class_Always_Null_OPEN).
Pipeline 24mo UI показывал 'Не указан: 4075 квартир'.

Fix: SQL COALESCE с correlated subquery через objective_complex_mapping →
objective_lots.class. Mapped complexes (114/4548) теперь резолвят корректный
класс — комфорт/стандарт/бизнес/премиум.

## Coverage impact (verified via MCP)

Pre-fix: 0 of 1316 planned ЖК имели obj_class
Post-fix: 339 of 1316 (25.8%) — 249 комфорт, 45 стандарт, 42 бизнес

Не-mapped objects остаются 'Не указан' (upstream kn-scraper fix всё ещё open).

Closes #162

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 10:07:52 +03:00
lekss361
46584d7c75
fix(velocity): :window_interval::interval cast syntax — same bug as PR #152 (#160)
SQLAlchemy text() parser confuses ':window_interval' (named param) followed
by '::interval' (cast operator) — exactly как было в PR #152 с :weights::jsonb.

Result: psycopg видит literal ':window_interval' в SQL → syntax error →
exception caught by velocity catch → return None → UI shows null.

## Fix

':window_interval::interval' → 'CAST(:window_interval AS interval)' (2 мест:
sales_rows query + _get_ekb_median percentile).

Pre-push code-reviewer должен был catch — добавим в feedback что для каждого
text() SQL grep ':[a-z]*::' before push.

Refs: PR #158 deploy verify, e2e velocity = null root cause

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 09:46:01 +03:00
lekss361
8fa1d005fd
feat(velocity): migrate D2 sales source → objective_corpus_room_month (#158)
* feat(velocity): migrate D2 sales source from stale sale_graph to objective_corpus_room_month

Closes #156

* fix(velocity): cross-stack contract + NULL safety per pre-push review

3 pre-push code-reviewer findings fixed:

1. TS union extended ('objective'|'sale_graph'|'rosreestr_fallback')
   + UI conditions handle both objective и sale_graph как valid sources.
2. COALESCE(deals_total_vol_m2, deals_total_count * 45.0) — NULL safety
   for DKP-only rows (vol_m2 nullable, count > 0).
3. room_bucket parking filter — verified false positive (все 5 buckets
   apartments: студия/1/2/3/4+).

Refs: PR #157 pre-push code-reviewer

* fix(types): add 'objective' to velocity_source union (cherry-pick miss)

Cherry-pick от закрытого PR #157 потерял этот файл. Frontend type-check fails
с 'no overlap' error. Restoring.

Refs: PR #158 frontend CI failure

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 09:34:00 +03:00
lekss361
c39b53cf53
fix(site-finder): cascade InFailedSqlTransaction — velocity internal rollback (#155)
* fix(site-finder): db.rollback() в velocity catches + SAVEPOINT zoning/success_rec

velocity.py 3 internal try/except возвращали None БЕЗ db.rollback(). PR #154
SAVEPOINT не помог — exception не propagates из velocity. Postgres tx
aborted внутри, cascade InFailedSqlTransaction на следующих queries.

Fix:
- velocity.py: db.rollback() в 3 catch блоках
- parcels.py: SAVEPOINT обёртки для zoning + success_recommendation

Refs: user reports 2026-05-15 cascade 500

* fix(velocity): SAVEPOINT per query (not db.rollback) — correct SQLAlchemy 2.0 pattern

Per PR #155 bot review — мой db.rollback() в catches был НЕПРАВИЛЬНЫЙ.

Session.rollback() = ENTIRE outer tx (не savepoint). Outer SAVEPOINT context
становится orphaned → InvalidRequestError на __exit__.

Correct fix: wrap каждый db.execute в свой with db.begin_nested():
- Failure → __exit__ propagates exception → SAVEPOINT rolls back
- Outer tx остаётся clean
- velocity returns None gracefully → caller продолжает

Все 3 SQL queries в velocity.py теперь SAVEPOINT-wrapped:
- compute_velocity: competitor_query
- compute_velocity: sales_rows query
- _get_ekb_median: median query

db.rollback() removed из всех 3 catches.

Refs: PR #155 bot review

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 08:52:38 +03:00
lekss361
b13c4be0ed
fix(parcels): SAVEPOINT around velocity compute → не abort outer tx (#154)
Production POST /parcels/{cad}/analyze падает с 500:
  sqlalchemy.exc.InternalError: (psycopg.errors.InFailedSqlTransaction)
  current transaction is aborted, commands ignored until end of transaction block

## Root cause

`compute_velocity()` SQL fails (probably на cad's без conkurrentов / sparse
sale_graph data) → exception caught в try/except → НО db.rollback() отсутствует
→ transaction остаётся в aborted state.

Следующая query (_geotech_risk на line 828) пытается выполниться → крашится
с InFailedSqlTransaction.

## Fix

Wrap velocity в `with db.begin_nested()` — SAVEPOINT pattern (consistent
with PR #124 pzz_loader fix). Failure внутри savepoint:
- Rollbacks ТОЛЬКО savepoint
- Outer transaction остаётся clean
- Subsequent queries (_geotech_risk и пр.) работают

Pattern matches feedback_subagent_delegation audit recommendation для
in-loop / per-section exception handling.

## Impact

POST /parcels/{cad}/analyze больше не 500 при velocity failure. Возвращает
`velocity: null` + остальные fields normal.

Refs: user report 2026-05-15 InFailedSqlTransaction

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 08:29:24 +03:00
lekss361
d0b8eda6ad
fix(deploy): source root .env (где POSTGRES_USER) не backend/.env (#153)
PR #151 deploy упал с 'POSTGRES_USER: unbound variable'.

source backend/.env — backend/.env содержит app vars (SCRAPE_ADMIN_TOKEN,
OPENROUTESERVICE_API_KEY и пр.), НО НЕ POSTGRES_USER/PASSWORD/DB.

Postgres credentials живут в /opt/gendesign/.env (root) — оттуда их читает
compose через env_file для postgres service. Это canonical location для
DB creds на этом стеке.

Fix: source .env (root) — corresponds к compose env_file resolution.

После merge → next deploy auto-apply применит pending #89/#91 (90 уже
applied via MCP).

Refs: #150, incident 2026-05-15 deploy #25901378924

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 08:19:17 +03:00
lekss361
af39e95f83
fix(site-finder): SQLAlchemy syntax error :weights::jsonb → CAST(:weights AS jsonb) (#152)
Production POST /weight-profiles → 500 SyntaxError.

## Root cause

SQLAlchemy text() parser трактует ':weights' followed by '::jsonb' двусмысленно:
- ':weights' — named param binding (должен стать %(weights)s)
- '::jsonb' — PG cast operator

Parser НЕ распознаёт :weights как параметр когда сразу следует :: cast.
Result: psycopg видит literal ':weights' в SQL → syntax error.

```
LINE 5: ($1, $2, :weights::jsonb, $3, $4)
                   ^
parameters: {user_id, profile_name, is_default, description}
            ↑ note: 'weights' missing — SQLAlchemy skipped it
```

## Fix

Replace ambiguous :weights::jsonb с CAST(:weights AS jsonb):
- _INSERT (line 102)
- _UPDATE dynamic 'weights = :weights::jsonb' (line 273)

CAST() syntax не имеет ambiguity, SQLAlchemy корректно bind'ит :weights.

## Affected endpoints

- POST /api/v1/admin/site-finder/weight-profiles (create)
- PUT /api/v1/admin/site-finder/weight-profiles/{id} (update with weights)

Refs: user report 2026-05-15

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 08:12:18 +03:00
lekss361
7c43bc5496
feat(infra): auto-apply data/sql/*.sql migrations in deploy pipeline (#150) (#151)
* fix(frontend): apiFetchWithStatus body stream double-read crash

Per user report 2026-05-14: 'Failed to execute text on Response: body stream
already read' при analyze с non-JSON error response.

apiFetchWithStatus делал .json() (consumes stream), потом .text() в catch
(FAILS — stream already consumed). Fetch API: body — ReadableStream,
consumable один раз.

Fix: read text() once → JSON.parse on string. На fail (HTML / non-JSON)
оборачиваем в {detail: rawText}. Никаких double-reads.

Site-finder cad search больше не крашится на error pages.

* feat(infra): auto-apply data/sql/*.sql migrations in deploy pipeline (#150)

Закрывает root cause production 500 на /api/v1/admin/site-finder/weight-profiles
(а также 2 unapplied migrations: #89/#91).

## Why это нужно

После audit обнаружили что:
- backend/alembic/versions/ пустой (Alembic configured но не используется)
- deploy.yml не имеет migration step
- 47+ raw SQL files накопилось без auto-apply
- 3 файла unapplied: 87 (engineering), 89 (drop brin), 90 (weight profiles)

User report: GET weight-profiles → 500 потому что user_weight_profiles
table не существует на prod.

## Changes

### deploy.yml +39 lines
- Path trigger расширен на data/sql/*.sql
- New step 'Apply DB migrations' между compose pull и compose up -d:
  - Idempotent CREATE TABLE _schema_migrations
  - Loop по data/sql/*.sql sorted → skip applied → psql with ON_ERROR_STOP=on
  - Record applied filename в tracking table
  - Exit 1 на failure → containers НЕ обновляются (no partial state)

### NN collision fix
- 87_engineering_networks_191fz.sql → 91_engineering_networks_191fz.sql
  (collision с 87_on_demand_indexes.sql, мой renumber 85→87 был неудачным)

### CLAUDE.md +32 lines
- Subsection 'Auto-apply' под Migration pattern
- Bootstrap procedure docs
- Idempotency requirement
- Failure recovery

### scripts/bootstrap_schema_migrations.sh +100 lines
- One-time seed _schema_migrations с 48 already-applied files
- 3 файла intentionally NOT seeded: 89/90/91 (auto-apply на first deploy)
- Usage docs + verification query

## Vault

domains/infra/Runbook_Auto_Apply_Migrations.md NEW
infra-MOC.md updated

## Required manual steps ДО merge

1. SSH gendesign (tunnel)
2. POSTGRES creds export
3. bash scripts/bootstrap_schema_migrations.sh — seed 48 already-applied files
4. Verify: SELECT COUNT(*) FROM _schema_migrations → 48
5. После merge → first deploy auto-apply #89/#90/#91 → 500 closes

Closes #150

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 08:06:54 +03:00
lekss361
142ed95d36
fix(frontend): apiFetchWithStatus body stream double-read crash (#149)
Per user report 2026-05-14: 'Failed to execute text on Response: body stream
already read' при analyze с non-JSON error response.

apiFetchWithStatus делал .json() (consumes stream), потом .text() в catch
(FAILS — stream already consumed). Fetch API: body — ReadableStream,
consumable один раз.

Fix: read text() once → JSON.parse on string. На fail (HTML / non-JSON)
оборачиваем в {detail: rawText}. Никаких double-reads.

Site-finder cad search больше не крашится на error pages.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 07:39:21 +03:00
lekss361
088c1ab2cf
feat(site-finder): VelocityBlock в MarketTab (#34 sub-PR 2/2 FINAL) (#147)
* feat(site-finder): VelocityBlock в MarketTab (#34 sub-PR 2/2 FINAL)

D2 velocity-score frontend: score gauge, period meta, top 5 competitors table.
TS types (Velocity/VelocityCompetitor/VelocityPeriod) добавлены.
Guard "velocity" in data для backward-compat pre-#146.

tsc 0 errors, ESLint 0 warnings.

Closes #34

* fix(site-finder): align VelocityBlock TS types with backend response shape

Per #147 bot review — cross-stack contract mismatch:

Backend velocity.py (PR #146 merged) returns:
  obj_id, name, dev_name, obj_class, district_name, distance_m, total_sqm_period

Frontend types had:
  obj_id, name, class, distance_m, total_sqm

Result: c.class undefined → never showed class chip; c.total_sqm undefined →
Math.round(NaN) = NaN, 'не число' в table column для всех 5 строк.

## Fix

- VelocityCompetitor.class → obj_class
- VelocityCompetitor.total_sqm → total_sqm_period
- Added optional dev_name, district_name fields
- Updated VelocityBlock.tsx — все 3 references (guard, chip render, table cell)

Refs: #34, #147

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 01:44:16 +03:00
lekss361
e93bb6121e
feat(site-finder): D2 velocity-score from domrf_kn_sale_graph (#34 sub-PR 1/2) (#146)
compute_velocity service queries competitor sales в радиусе 3км:
- ST_MakePoint(longitude, latitude) — domrf_kn_objects не имеет geom column
- JOIN domrf_kn_sale_graph за 6 мес (area_sq primary, realised*45 fallback)
- Normalize vs ЕКБ-wide median → velocity_score 0..1
- confidence: high/medium/low (competitors_count + months_observed)
- Top 5 sample competitors для UI

Integration: analyze_parcel.response['velocity'] top-level field.

Schema corrections vs spec:
- obj_name → comm_name
- region_code → region_cd
- contracted (INT) → area_sq (м²)

Tests: 102/102 pass.
Vault: Module_Velocity_Service.md NEW.

Closes #144 (sub-PR 2 frontend закроет #34)

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 01:24:53 +03:00
lekss361
95647a1c82
feat(site-finder): G5 gate verdict banner in OverviewTab (#32 sub-PR 2/2 FINAL) (#143)
GateVerdictBanner с 4 visual states + expand/collapse details.
Types: GateVerdict/GateBlocker/GateWarning/GateVerdictSource/GateVerdictLabel.
Renders at top of OverviewTab driven by backend aggregator (#142).

tsc 0 errors, ESLint 0 warnings.

Vault: code/modules/Module_Gate_Verdict_Banner.md NEW.

Closes #32

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 01:09:34 +03:00
lekss361
76a88b2827
feat(site-finder): gate verdict aggregator — can-build-MKD signal (#32 G5 sub-PR 1/2) (#142)
Pure-function aggregator collapses nspd_zoning / nspd_zouit_overlaps /
nspd_engineering_nearby / nspd_dump в один GateVerdict TypedDict.

Logic:
- ПЗЗ не Ж-* → BLOCKER
- ЗОУИТ sub=17 (инжен. охранная) → BLOCKER
- Другие ЗОУИТ → WARNING
- Нет инжен. сетей в 200м → WARNING
- nspd_dump stale → source: nspd_dump_partial
- Нет dump → can_build_mkd: 'unknown'

Integration: новое поле gate_verdict в analyze_parcel response.

Tests: 15/15 pass (mock-based).

Vault: code/modules/Module_Gate_Verdict.md NEW.

Closes #140 (sub-PR 2 frontend закроет #32)

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 00:59:53 +03:00
lekss361
b42c9dd8f5
feat(site-finder): weight profile panel + analyze options (#114 sub-PR 4/4 FINAL) (#139)
Closing FINAL sub-PR for #114 Custom POI weights — UI complete.

- weightProfiles.ts: TS types + 4 TanStack Query hooks + 11 categories
- WeightProfilePanel.tsx: collapsible panel, sliders, save dialog
- useSiteAnalysis.ts: accepts AnalyzeOptions, query string for backend
- page.tsx: lifts weights state, integrates panel

Approach: 'save first' для ephemeral (no temp profile creation).

tsc 0 errors, ESLint 0 warnings.

Vault: Module_Weight_Profiles_Frontend.md NEW.

Closes #114

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 00:50:02 +03:00
lekss361
7250aa6187
feat(site-finder): weight profiles endpoints + analyze integration (#114 sub-PR 3/4) (#138)
Admin CRUD router + non-breaking analyze_parcel integration. 46 tests pass.

Refs: #114

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 00:40:06 +03:00
lekss361
5aea78c2d8
feat(site-finder): weight profiles service — Pydantic + CRUD (#114 sub-PR 2/4) (#137)
Per #114 sub-PR 2/4. Pydantic v2 models + CRUD service. 14 passing tests.

API: list/get/get_default/create/update/delete_profile + resolve_weights
fallback to system defaults.

Validation: ALLOWED_CATEGORIES guard + weight bounds [-2.0, 3.0].
Default-uniqueness в одной transaction (job_settings.py pattern).

Vault: code/modules/Module_Weight_Profiles_Service.md.

Refs: #114

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 00:29:37 +03:00
lekss361
f3f1e7f6bc
feat(db): schema user_weight_profiles (#114 sub-PR 1/4 — foundation) (#136)
Per #114 Custom POI weights — Макс feedback "садики и Мегамарт должны
иметь разные веса". Sub-PR 1/4 per feedback_split_big_issues:

1. **#114 sub-PR 1 (this)**: schema migration
2. #114 sub-PR 2: backend Pydantic models + CRUD service
3. #114 sub-PR 3: POST /parcels/{cad}/analyze?profile= override + admin endpoint
4. #114 sub-PR 4: frontend 25 sliders UI + live preview

## Schema

`data/sql/90_user_weight_profiles.sql`:

- Table `user_weight_profiles` — SERIAL PK + user_id + profile_name +
  weights JSONB + is_default + timestamps
- Unique constraint (user_id, profile_name)
- B-tree index user_idx
- Partial index `WHERE is_default = TRUE` (fast default lookup)
- Trigger auto-updated_at via `set_updated_at()` function

## Compat with existing POI categories

`weights` JSONB compatible с current `_POI_WEIGHTS` const в
`backend/app/api/v1/parcels.py`:
- school 1.5, kindergarten 1.5, pharmacy 0.8, hospital 0.6
- shop_mall 1.2, shop_supermarket 1.0, shop_small 0.5
- park 1.8, bus_stop 0.3, metro_stop 1.5, tram_stop -0.5

## Idempotency

- BEGIN/COMMIT обёртка
- IF NOT EXISTS на CREATE TABLE/INDEX
- DROP TRIGGER IF EXISTS → CREATE TRIGGER (safe re-run)
- CREATE OR REPLACE FUNCTION (idempotent)

## Apply

```bash
psql postgresql://gendesign:...@localhost:15432/gendesign \
  -f data/sql/90_user_weight_profiles.sql
```

После apply таблица inert (нет backend code yet). Sub-PR 2 добавит CRUD.

## Vault

`code/schemas/Schema_User_Weight_Profiles.md` — created (status:
ready_to_apply, full sub-PR plan).

Refs: #114

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 00:19:33 +03:00
lekss361
6a256bfc1f
refactor(workers): split celery_app.py god-object into 3 modules (#135)
Per audit batch #127 P3 god objects (issue #134).

Split layout:
- celery_app.py: 270 → 35 lines (config only)
- beat_schedule.py: 207 lines NEW (DB-driven + fallback + hardcoded)
- lifecycle.py: 168 lines NEW (worker_process_init + worker_ready handlers)

No business logic changes — move-only structural refactor. Public API
(celery_app.conf.beat_schedule) unchanged.

Verify:
- beat keys: kn-region-66, objective_sync, refresh-ekb-districts-medians,
  nspd-geo-zombie-cleanup, poi-sync-weekly, noise-sync-weekly
- ruff + ruff-format clean
- AST parse OK

Vault: Module_Beat_Schedule + Module_Worker_Lifecycle NEW;
Module_Celery_App updated.

Closes #134
Refs: #127

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 00:09:38 +03:00
lekss361
2416ed0db1
refactor(frontend): extract SectionLabel + EmptyState + adminStyles shared (#132)
* fix(docs): block javascript: URLs in renderMarkdown href (XSS guard)

Per audit batch #127 P2 security minor (issue #130).

## Risk

`renderMarkdown` substituted `[text](url)` → `<a href="...url...">` verbatim
без URL validation. Edge case: `[click](javascript:alert(1))` →
`<a href="javascript:alert(1)">click</a>` execute'нется при click.

Currently low risk (markdown source = build-time fs.readFileSync из
public/docs/, checked into repo), но защита-в-глубину: если когда-либо
markdown source станет user-supplied, vuln materialize.

## Fix

`frontend/src/app/docs/b2b-channels/renderMarkdown.ts`:

1. New `safeUrl(url)` function — allowlist `https://`, `http://`, `/`, `#`,
   `mailto:`. Everything else → `"#"`.
2. Link regex replacement switched from backreference string to callback
   so URL passes через `safeUrl` ДО injection в href.
3. URL unescape → safeUrl → re-escape (HTML escape applied per-line).

## Test cases verified

- `[ok](https://example.com)` → href=`https://example.com` 
- `[ok](/local)` → href=`/local` 
- `[ok](#anchor)` → href=`#anchor` 
- `[ok](mailto:foo@bar.com)` → href=`mailto:foo@bar.com` 
- `[bad](javascript:alert(1))` → href=`#` 
- `[bad](data:text/html,...)` → href=`#` 
- `[bad](vbscript:msgbox(1))` → href=`#` 

## Checks

- tsc: 0 errors
- lint: 0 warnings
- No existing `__tests__/renderMarkdown` to break

## Vault

`fixes/Bug_RenderMarkdown_JavascriptUrl_May14.md` — created.

Closes #130

* refactor(frontend): SectionLabel + EmptyState + adminStyles shared (fixup)

Per audit batch #127 P2 hygiene (issue #129). Previous commit лошил
файлы из-за pre-commit (видимо). Fixup-коммит с actual diff:

## New shared modules (3)

- frontend/src/components/ui/SectionLabel.tsx
- frontend/src/components/ui/EmptyState.tsx
- frontend/src/lib/adminStyles.ts

## Replacements

- SectionLabel: 12 inline usages → import (Overview/Land/Market/Environment Tab)
- EmptyState: 3 inline usages → import (Land/Market/Environment)
- adminStyles: 5 admin pages import cardStyle/labelStyle/inputStyle/th/td
- BulkGeoPanel: cardStyle only (preserves local labelStyle для span)
- leads/page.tsx: cardStyle с marginTop extend; th/td/inputStyle local (divergent)

## Visual regression: ZERO

All px/colors/CSS properties copied verbatim. Files с divergent values left local.

## Checks

- ruff (no files)
- prettier
- tsc + lint clean (per agent run)

Closes #129
Refs: #127

* docs(adminStyles): JSDoc warning про divergent local overrides

Per bot review #132 non-blocking minor — задокументировать почему
leads/page.tsx и BulkGeoPanel.tsx имеют local overrides, чтобы будущий
читатель не «унифицировал» обратно по ошибке.

Refs: #129, #132

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:55:38 +03:00
lekss361
389fd31fad
refactor(admin): extract _check_token into shared AdminTokenAuth dep (#133)
Per audit batch #127 P2 hygiene (issue #128).

## Problem

Identical `_check_token` function в 3 admin files:
- backend/app/api/v1/admin_scrape.py:37-44
- backend/app/api/v1/admin_leads.py:26-33
- backend/app/api/v1/admin_jobs.py:24-31

## Fix

New `backend/app/core/deps.py` с `verify_admin_token` + Annotated alias:
```python
AdminTokenAuth = Annotated[None, Depends(verify_admin_token)]
```

Заменено в 27 endpoints (18 admin_scrape + 6 admin_leads + 3 admin_jobs):
- Removed local `_check_token` definitions
- `_: None = Depends(_check_token)` → `_: AdminTokenAuth`
- Removed unused imports (`Header`, `HTTPException`, `settings` где не используется)

## Semantics preserved

Все 3 original implementations были identical (503 if env missing + 401 if
mismatch, same header `X-Admin-Token`, same env `SCRAPE_ADMIN_TOKEN`).

Минор: 503 detail text unified to `"admin disabled — set SCRAPE_ADMIN_TOKEN"`
(в оригинале три разные строки — `"admin scrape disabled..."` etc).
Cosmetic — detail в HTTP 503 не парсится клиентским кодом.

## Tests

`uv run pytest tests/ -x -k admin` → 64/64 passed.

## Vault

`code/patterns/Pattern_Admin_Token_Dependency_May14.md` — created.

Closes #128
Refs: #127

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:45:44 +03:00
lekss361
8647114b22
fix(docs): block javascript: URLs in renderMarkdown href (XSS guard) (#131)
Per audit batch #127 P2 security minor (issue #130).

## Risk

`renderMarkdown` substituted `[text](url)` → `<a href="...url...">` verbatim
без URL validation. Edge case: `[click](javascript:alert(1))` →
`<a href="javascript:alert(1)">click</a>` execute'нется при click.

Currently low risk (markdown source = build-time fs.readFileSync из
public/docs/, checked into repo), но защита-в-глубину: если когда-либо
markdown source станет user-supplied, vuln materialize.

## Fix

`frontend/src/app/docs/b2b-channels/renderMarkdown.ts`:

1. New `safeUrl(url)` function — allowlist `https://`, `http://`, `/`, `#`,
   `mailto:`. Everything else → `"#"`.
2. Link regex replacement switched from backreference string to callback
   so URL passes через `safeUrl` ДО injection в href.
3. URL unescape → safeUrl → re-escape (HTML escape applied per-line).

## Test cases verified

- `[ok](https://example.com)` → href=`https://example.com` 
- `[ok](/local)` → href=`/local` 
- `[ok](#anchor)` → href=`#anchor` 
- `[ok](mailto:foo@bar.com)` → href=`mailto:foo@bar.com` 
- `[bad](javascript:alert(1))` → href=`#` 
- `[bad](data:text/html,...)` → href=`#` 
- `[bad](vbscript:msgbox(1))` → href=`#` 

## Checks

- tsc: 0 errors
- lint: 0 warnings
- No existing `__tests__/renderMarkdown` to break

## Vault

`fixes/Bug_RenderMarkdown_JavascriptUrl_May14.md` — created.

Closes #130

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:39:47 +03:00
lekss361
8a266502a1
fix(sql): wrap 73_complexes_master.sql in BEGIN/COMMIT (atomic apply) (#125)
Per code review audit (May 14): файл без transaction wrapper — backfill CTE
INSERTs + view recreations executed как individual auto-commit statements.
Mid-file crash → orphan rows в complexes/complex_sources с no rollback path.

## Fix

`BEGIN;` после header comments (line 31, перед section 1).
`COMMIT;` в конце файла после последнего COMMENT ON VIEW.

Wraps все 6 секций atomically:
1-2: CREATE TABLE + CREATE INDEX для complexes + complex_sources
3-5: 3 backfill CTE INSERTs с ON CONFLICT DO NOTHING
6: DROP VIEW IF EXISTS + CREATE VIEW + COMMENT для v_complex_full

## No CONCURRENTLY split needed

Проверил — все 10 indexes используют plain `CREATE INDEX IF NOT EXISTS` без
CONCURRENTLY keyword. Безопасно обернуть в одну tx.

## Idempotency preserved

Все existing guards остаются:
- CREATE TABLE/INDEX IF NOT EXISTS
- ON CONFLICT (source, source_id) DO NOTHING
- DROP VIEW IF EXISTS

Re-running file двукратно — safe.

## Vault

`fixes/Migration_73_Complexes_Atomic_May14.md` — created (status: resolved).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:29:28 +03:00
lekss361
a3e7d21e3a
fix(pzz_loader): replace bare db.rollback() with SAVEPOINT per-row (#124)
Per code review audit (May 14): transaction correctness bug в
`sync_pzz_zones_to_db()` — inner `db.rollback()` в loop сбрасывал outer
transaction на per-row failures → counts inserted/updated НЕ matched DB state.

## Fix

Wrap UPSERT в loop через `db.begin_nested()` (SAVEPOINT):
- Failure одной row → rollback только её savepoint, outer tx alive
- Successfully inserted rows accumulate в outer tx → commit final
- Outer try/except добавлен для unexpected error → rollback + log + raise
- `finally: db.close()` preserved

Pattern consistent с `domrf_kn.py` (где SAVEPOINT уже работает).

## Preserved

- Function signature + return shape {fetched, inserted, updated, skipped} — unchanged
- Logging contracts
- skipped count теперь корректно включает DB-failure rows (consistent with prior behavior)

## Tests

- No test_pzz_loader.py — skipped
- ruff + AST passed

## Vault

`fixes/Bug_Pzz_Loader_Missing_Savepoint_May14.md` — created (status: resolved).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:29:23 +03:00
lekss361
8dfab7613d
fix(nspd-geo): replace f-string bulk INSERT with parametrized batch execute (#123)
Per code review audit (May 14): real SQL injection vector в `enqueue_geo_job`.
Manual `chr(39)*2` quote escape НЕ injection-safe — не защищает от backslash
escape, Unicode quote tricks. Если cad_num когда-либо придёт из user input —
exploitable.

## Change

`backend/app/workers/tasks/nspd_geo.py:298-310` — заменил f-string VALUES
concatenation на SQLAlchemy `text(...)` + list of dicts (psycopg v3 handles
batch executemany под капотом).

Pattern consistent с остальными INSERT'ами в этом же файле (_log,
_start_job, _save_quarter, _save_parcel, _save_building).

## Preserved

- ON CONFLICT (job_id, cad_num, thematic_id) DO NOTHING — verbatim
- Return value (int(job_id))
- Surrounding tx/commit/close logic
- DB schema, function signature, caller interface — unchanged

## Tests

- No test_nspd_geo.py существует — skipped
- ruff check passed

## Vault

`fixes/Bug_Nspd_Geo_Sql_Injection_May14.md` — created (status: resolved).

No backward compat / migration concerns — pure in-place query rewrite.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:29:12 +03:00
lekss361
7b32edc34d
fix(db): DROP dead BRIN index rosreestr_deals_period_brin (#122)
Per code review audit (May 14): BRIN index на partitioned parent table dead
weight. PG16 НЕ propagates такие индексы на child partitions; pg_stat
показывает idx_scan=0 → никогда не использовался.

## Why dead

rosreestr_deals — yearly partitioned по period_start_date. Queries делают
partition pruning через partition key. BRIN index на parent существует, но:
1. На child partitions его нет (PG16 не reflects)
2. Parent сам не имеет данных (partitioned out)
3. Queries никогда не trigger BRIN scan

Maintenance overhead (storage, autoanalyze) без value.

## Migration

`data/sql/89_drop_dead_brin_rosreestr_deals.sql` — атомарный DROP с idempotent
`IF EXISTS`. Apply order: после 88_*.

## Cleanup

`data/sql/01_schema_rosreestr_deals.sql:65-67` — закомментировал оригинальное
CREATE с note: fresh DB bootstrap НЕ создаст index заново.

## Rollback path

Если в будущем нужен range scan optimization — re-create per-partition:
```sql
CREATE INDEX rosreestr_deals_2024q1_period_brin
  ON rosreestr_deals_2024q1 USING BRIN (period_start_date);
-- per-partition × N partitions
```

Но обычно partition pruning достаточно. Измерь EXPLAIN перед re-creation.

## Vault

`fixes/Migration_Drop_Dead_Brin_May14.md` создан (status: ready_to_apply).

Apply на prod: `psql ... -f data/sql/89_drop_dead_brin_rosreestr_deals.sql`
после merge.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:29:01 +03:00
lekss361
f5794173de
fix(celery): disable nspd-harvest-stale-quarters beat (Bug post-mortem #1) (#121)
Per fixes/Bug_NSPD_WMS_NotBulk_2026_May14 action item #1: отключить beat
schedule `nspd-harvest-stale-quarters` до Sprint 2 fix (grid sampling rewrite).

## Why

harvest_quarter в текущей реализации пишет почти пустые dumps в
nspd_quarter_dumps из-за single-pixel WMS GetFeatureInfo bug (детали в
vault post-mortem). Beat запуск Mon 04:00 МСК (next: 2026-05-18 04:00 МСК):

- Потратит rate-limit budget на ~12k EKB кварталов
- Заполнит nspd_quarter_dumps мусорными rows
- analyze_parcel начнёт отдавать пустые nspd_zoning/nspd_zouit_overlaps/
  nspd_engineering_nearby для всех пользователей (через quarter_dump_lookup)

## What changed

`backend/app/workers/celery_app.py` — закомментирован `schedule[
"nspd-harvest-stale-quarters"]` блок (~10 строк) с inline-комментом
explanation + ссылкой на vault post-mortem.

## What НЕ changed

- `backend/app/workers/tasks/nspd_sync.py` — task code (harvest_quarter,
  harvest_stale_quarters) остаётся, можно re-enable после Sprint 2 fix
- Admin endpoint для manual trigger остаётся работающим
- Все остальные beat tasks (poi-sync, noise-sync, zombie-cleanup,
  refresh-ekb-districts-medians, kn-region-*) — не тронуты

## Vault

- fixes/Bug_NSPD_WMS_NotBulk_2026_May14.md — status `identified` → `mitigated`
  (уже обновлено через vault audit pass)

## Re-enable

После Sprint 2 grid-sampling rewrite (см. Bug post-mortem → Sprint 2
fix-strategy) — uncomment блок + verify через staging.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:24:53 +03:00
lekss361
b3f32ae927
chore(infra): correct frontend healthcheck + tighten depends_on (re-apply #117) (#120)
Forward-fix после incident May 14 (см. vault events/Incident_Prod_Down_May14_2026).
PR #117 был полностью reverted PR #119 (Caddy depends_on frontend: service_healthy
+ failing wget --spider /  → Caddy не стартовал → vsё прод down ~25 мин).

Этот PR делает то же что планировал #117, но корректно:

### Frontend healthcheck — TCP probe через node (НЕ HTTP)

```yaml
test: ["CMD", "node", "-e",
  "require('net').createConnection({port:3000,host:'127.0.0.1'})
   .once('connect',function(){process.exit(0)})
   .once('error',function(){process.exit(1)})"]
start_period: 60s
retries: 6
interval: 15s
```

- НЕ зависит от HTTP status (200/404/308 — все OK если порт слушает)
- `node` гарантированно в alpine image (запускает server.js)
- `127.0.0.1` — никакой DNS ambiguity (busybox alpine localhost IPv6 vs IPv4)
- 60s + 6×15s = 150s window — с запасом на Next.js standalone cold start

### Caddy depends_on — backend healthy, frontend started (НЕ healthy)

```yaml
caddy:
  depends_on:
    backend: { condition: service_healthy }    # backend имеет /health endpoint
    frontend: { condition: service_started }   # safety net — НЕ блокирует Caddy
```

Lesson from incident: frontend health flaky (Next.js cold start variable).
Strict service_healthy для frontend → одна misconfig = full outage обоих доменов
(gendsgn.ru + obsidian.gendsgn.ru через тот же Caddy на shared network).

### backend/worker/beat redis: service_started → service_healthy

Redis уже имеет `redis-cli ping` healthcheck. Tightening безопасно — был
artifact pre-healthcheck Redis.

### Pin rosreestr2coord>=5.0.0 (lock уже 5.3.3)

v5 убрала delay параметр; код адаптирован (rate limit на worker уровне).
Защита от silent downgrade при `uv lock --upgrade`.

### Что НЕ повторено из #117

- Caddy depends_on frontend: service_healthy — заменено на service_started
- wget --spider / — заменено на node TCP probe

### Vault обновлён

- `events/Incident_Prod_Down_May14_2026.md` — постмортем + timeline + lessons
- Update Changelog в `domains/infra/infra-MOC.md` (если есть)

Closes incident. Re-applies #117 goals safely.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:24:47 +03:00
lekss361
6ea2dcd46c
Revert "chore(infra): healthcheck frontend + tighten depends_on; pin rosreestr2coord>=5 (#117)" (#119)
This reverts commit bade2f7772.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 23:02:58 +03:00
lekss361
d4cb56d093
fix(sql): renumber engineering_networks 85→87 + dedup classid + add SRID 4326 (#118)
* fix(sql): renumber engineering_networks 85→87, dedup classid, add SRID

Three critical fixes на untracked 85_engineering_networks_191fz.sql per
code review (May 14) — до первого применения на prod.

1. **NN collision fix**: renumber 85_engineering_networks_191fz.sql → 87_
   85_pzz_zones_ekb.sql уже занимает 85_ prefix. При alphabetic deploy
   order `85_e*` применился бы ДО `85_p*` — но 87_ ставит нас после
   86_v_bucket_success_score.sql без зависимости на 85_pzz.

2. **Duplicate classid '907015504'**:
   - Был дважды: 'Опора металлическая' + 'Опора железобетонная'.
   - Второй ON CONFLICT DO UPDATE молча перетёр бы первый → NSI код
     металлической опоры терялся бы.
   - Fix: 907015504 остаётся за металлической (lower-numbered subtype
     per Минстрой NSI source), железобетонная переезжает на 907015507
     (заполняет gap 906→508). Помечено comment-меткой 'verify with
     Минстрой NSI table' до prod deploy.

3. **Untyped geometry**:
   - Было: `geom GEOMETRY` без SRID constraint → разрешало бы insert
     mix MSK-66 (srid=100000) и WGS84 (srid=4326) → silent ST_DWithin
     corruption.
   - Fix: `geom GEOMETRY(Geometry, 4326)`. Capital G позволяет
     mixed Point/LineString/Polygon, SRID 4326 жёстко фиксирует WGS84.
     ETL loader должен делать ST_Transform(msk66_raw, 4326) перед
     insert (см. backend/app/core/crs.py для CRS utility).

Idempotency: BEGIN...COMMIT, IF NOT EXISTS, ON CONFLICT DO UPDATE
паттерн — уже корректные в файле.

Closes Day-1 critical #2 from code review audit (May 14).
Pre-deploy: verify 907015507 classid assignment against Минстрой
official NSI table; reassign and re-run migration if needed.

* fix(sql): remove provisional 907015507, fix Фонати→Фонари typo

Addresses review blocker on PR #118:

- Provisional classid 907015507 ('Опора железобетонная (столб)') убран
  из seed insert. Commented out с TODO(NSI-verify) — следующий шаг
  верифицировать против официальной NSI-таблицы Минстроя ДО восстановления
  строки. Без верификации ON CONFLICT DO UPDATE name перетёр бы корректное
  значение в eng_ref_classid.
- Опечатка 'Фонати электрические' → 'Фонари электрические' (907015602).
- Sort: 904015504/505/506/508 теперь в численном порядке (после удаления
  провизорной 507).

Live seed теперь покрывает 504 (металл столб), 505 (металл ферма),
506 (металл фермовый столб), 508 (железобет ферма). Строка 507
restored after Минстрой NSI verification (см. follow-up).

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 22:51:11 +03:00
lekss361
bade2f7772
chore(infra): healthcheck frontend + tighten depends_on; pin rosreestr2coord>=5 (#117)
- Add wget healthcheck to frontend service (alpine-compatible) — Caddy
  больше не роутит трафик до того как Next.js готов.
- Caddy depends_on backend/frontend: service_started → service_healthy.
- backend/worker/beat redis dependency: service_started → service_healthy
  (Redis уже имеет redis-cli ping healthcheck).
- Pin rosreestr2coord>=5.0.0 (v5 убрала delay параметр; код уже адаптирован
  per CLAUDE.md); uv resolved 5.3.3.

Closes Day-1 quick wins #3+4 from code review audit (May 14).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-14 22:42:09 +03:00
lekss361
1c2f80a4b8
feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 FINAL) (#116)
* feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 #4 FINAL)

Замыкает Sprint 1.1 из #94 part 2 plan. После этого PR пользователь видит
свежие НСПД данные в UI (frontend integration — отдельный PR).

Backend (new app/services/site_finder/quarter_dump_lookup.py):
- `derive_quarter_cad(cad_num)` — 3/4/5-сегмент → quarter (3-segment)
- `get_quarter_dump_data(db, cad_num, parcel_wkt)` — main entrypoint:
  - Reads nspd_quarter_dumps row для derived quarter
  - Freshness threshold: 180 days
  - Missing/stale/harvest_error → trigger harvest_quarter.apply_async() fire-
    and-forget (lazy import против circular), return EMPTY_DUMP_RESULT
  - Fresh + parcel_wkt=None → metadata only (no spatial queries)
  - Fresh + geometry → 3 spatial queries via jsonb_array_elements + ST_Transform
    (3857→4326) + ST_Intersects / ST_DWithin
- 3 private helpers:
  - `_get_zoning` — point-in-polygon parcel centroid vs territorial_zones, LIMIT 1
  - `_get_zouit_overlaps` — все zouit_% layers пересекающиеся с parcel
  - `_get_engineering_nearby` — engineering_structures в 200m, sorted by distance
- `EMPTY_DUMP_RESULT` module-level constant — DRY для no-dump fallback (used
  in get_quarter_dump_data internal + analyze_parcel try/except wrap)

Backend (parcels.py):
- Import EMPTY_DUMP_RESULT + get_quarter_dump_data
- Call wrapped в try/except — если nspd_quarter_dumps недоступна (DB timeout
  / table missing) → EMPTY_DUMP_RESULT fallback вместо 500 (consistent с
  resilience pattern других optional fetches)
- Response gets 4 new fields:
  - nspd_zoning: dict | None (G1 ПЗЗ — zone_code, zone_name, source)
  - nspd_zouit_overlaps: list[dict] (G3 — overlaps в parcel, per ЗОУИТ group)
  - nspd_engineering_nearby: list[dict] (I3 — engineering structures в 200m)
  - nspd_dump: dict (freshness metadata — available, fetched_at_utc, stale,
    harvest_triggered, total_features)

Tests: 13 new в test_quarter_dump_lookup.py (mock-based, no real DB):
- derive_quarter_cad 5 edge cases (3seg, 4seg, 5seg, invalid, whitespace)
- get_quarter_dump no_row → harvest triggered
- stale (>180d) → harvest triggered, stale=True
- harvest_error row → retry harvest triggered
- parcel_wkt=None → metadata only (1 DB call)
- fresh + zoning extraction
- fresh + zouit_overlaps list
- fresh integration: все 4 keys present

47 pre-existing tests still pass.

Code review (code-reviewer pre-push): MINOR, 0 blocking. Applied 2 of 4:
-  #1: try/except wrap around get_quarter_dump_data в analyze_parcel
  (защита от DB unavailability) + DRY через EMPTY_DUMP_RESULT module const
-  #2: removed redundant nspd_zoning.fetched_at_utc (DRY — freshness в
  nspd_dump.fetched_at_utc)
- ⏭ Deferred (acceptable): #3 ad-hoc harvest_quarter retry cooldown для
  harvest_error rows (только при high traffic + persistent NSPD errors);
  #4 raw_props в response — tech debt, убрать вместе с frontend PR

Performance note: 3 spatial queries per analyze adds ~10-50ms on typical
~100-feature quarter. Mitigation if quarters grow dense: materialized
per-layer sub-table (отдельная DB issue).

Closes Sprint 1.1 part of #94. Frontend rendering этих 4 полей — отдельный
PR (next: #112 / #115 / #114).

* fix(site-finder): address PR #116 auto-review M1-M5

M1 (mutation risk): replace EMPTY_DUMP_RESULT direct refs with
_make_empty_result() factory. dict(...) shallow copy left nested
nspd_dump shared by reference across concurrent requests — single
mutation pollutes module sentinel for all subsequent calls. Now
каждый caller gets independent dict.

M2 (O(N) spatial scan): SELECT extended denormalized counts
(territorial_zones_count, zouit_count, engineering_count). Each
spatial helper accepts layer_counts and early-returns when count=0
— skips heavy jsonb_array_elements + ST_Transform/ST_Intersects
scan entirely. Critical для quarters с 2000+ features.

M4 (documentation): _trigger_harvest docstring describes known
burst/no-dedup limitation + TODO Redis SETNX (отдельный PR).

M5 (test fragility): _make_db_mock_with_spatial docstring describes
positional-call contract — db.execute order (0=dump, 1=zoning, 2=zouit,
3=engineering) и зависимость от count-values.

+4 new tests (17 total, all pass):
- test_make_empty_result_returns_independent_copies (mutation safety)
- test_make_empty_result_overrides
- test_early_exit_all_counts_zero_no_spatial_queries
- test_early_exit_partial_counts

Per auto-review on 3068a9c.

* fix(site-finder): rename _make_empty_result → make_empty_result (public) per PR #116 review

M1 residual fix: parcels.py exception path использовал EMPTY_DUMP_RESULT
singleton ref вместо factory. Сейчас readonly access, но нарушает
documented invariant модуля.

Rename `_make_empty_result` → `make_empty_result` (public API), import в
parcels.py, использовать в try/except fallback. Каждый request получает
независимый dict — никаких shared references.

M4 (Redis SETNX dedup) + M5 (test fragility) — deferred per review,
documented в code/issue. Acceptable trade-offs:
- M4: UPSERT idempotency делает данные safe; burst-duplicate task'и тратят
  WAF traffic впустую но не повреждают данные.
- M5: docstring contractually describes positional-call order.

17/17 tests pass. ruff/format clean.

Per auto-review on aef8308.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-13 09:14:19 +03:00
lekss361
f3b0ce5687
feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4 part A) (#111)
* feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4/4 part A)

Sprint 1.1 item #3 из #94 part 2. Замыкает harvest cycle: Celery task
оборачивает NSPDClient.search_by_quarter (PR #109) + UPSERT в
nspd_quarter_dumps (PR #110). После — analyze_parcel будет читать dumps
вместо on-demand NSPD HTTP в request-цикле (Sprint 1.1 #4, следующий PR).

Backend:
- New `backend/app/workers/tasks/nspd_sync.py`:
  - `harvest_quarter(quarter_cad, region_code=66, include_zouit=True,
    include_risks=False)` — single-quarter harvest. autoretry_for=
    (NspdLiteWafError,) с retry_backoff=True, max_retries=3,
    soft_time_limit=120s. Generic exception → upsert error row с
    harvest_error=str(exc), counts=0 (НЕ re-raise).
  - `harvest_stale_quarters(region_code, max_age_days=90, batch_size=50)` —
    beat fanout: SELECT cad_quarters_geom EXCEPT fresh dumps → apply_async per cad.
  - UPSERT использует ST_Multi(ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON,3857),4326))
    для quarter_geom (schema MultiPolygon strict per database-expert contract).
- `celery_app.py`:
  - Add `nspd_sync` в `include`
  - Beat entry `nspd-harvest-stale-quarters` Mon 04:00 МСК, batch_size=50,
    max_age_days=90
  - `worker_ready` sanity check `SELECT 1 FROM nspd_quarter_dumps LIMIT 0`
    (non-fatal critical log per Bug_Worker_Ready_EarlyReturn_Fixed pattern)
- `admin_scrape.py`:
  - `HarvestQuarterRequest` Pydantic с pattern `^\d+:\d+:\d+$` (quarter only)
  - `POST /api/v1/admin/scrape/nspd/harvest-quarter` manual trigger,
    requires X-Admin-Token, returns task_id.

Tests (`test_nspd_sync.py`): 14 mock-based (no real NSPD/DB):
- happy path: search_by_quarter mock → verify SQL params/structure
- empty quarter → row with quarter_geom=NULL, total=0, no harvest_error
- WAF error → autoretry kicks in (raise re-propagated)
- Generic error → upsert error row
- Fanout: 50 stale → 50 apply_async calls
- Layer names preservation (all 5 zouit + 11 risks if enabled)
- Geometry SRID handling

Code review (code-reviewer pre-push): APPROVE, 0 blocking, 5 minor deferred:
- inline datetime import in error branch (style)
- layers_fetched array literal via string concat (current layer names safe)
- harvest_stale_quarters error rows always re-tried (intended, undocumented)
- test call_args.kwargs fragile (works for current callsite)
- beat collision with refresh-ekb-districts-medians every ~2 years (negligible)

Part of #94. Next: Sprint 1.1 #4 — analyze_parcel reads dump cache.

* fix(nspd-sync): address PR #111 auto-review M1-M5+M7 findings

M1 (runtime bug): remove WHERE region_code from outer cad_quarters_geom SELECT
in harvest_stale_quarters — column does not exist in prod schema
(cad_number, geom, raw_props, fetched_at, source). Would crash on first beat
tick. Inner nspd_quarter_dumps subquery keeps region_code filter unchanged.

M2 (correctness): _upsert_dump теперь делает db.rollback() в except перед
db.close() — prevents PendingRollbackError на следующем session use, если
PostGIS отбракует geometry или JSON cast failure.

M3 (low): worker_ready breadcrumb INSERT тоже теперь имеет rollback —
consistency с остальными session usages в celery_app.py.

M4 (style): layers_fetched — заменил manual string concat
'{' + ','.join(...) + '}' на native Python list. psycopg v3 сам сериализует
list → text[]. Защита от future layer names с запятыми/фигурными скобками.

M5 (style): import datetime as _dt поднят из else branch в top-level imports.

M7 (test cleanup): удалил dead __enter__/__exit__ mock setup в
test_harvest_stale_quarters_fanout / _empty. harvest_stale_quarters не
использует session как context manager — moc'и no-op.

14/14 tests pass. ruff/format clean.

Per auto-review on 9b2a289.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 18:45:57 +03:00
lekss361
6fce81e10c
feat(db): add nspd_quarter_dumps table for PKK harvest persistence (#94 pt.3/4) (#110)
Sprint 1.1 item #2 из плана #94 part 2. Persistent storage для QuarterDump
snapshots (PR #109). Foundation для Celery harvest_quarter task (next PR)
и analyze_parcel integration (после).

Schema (data/sql/88_nspd_quarter_dumps.sql):
- Table `nspd_quarter_dumps`: 17 columns matching QuarterDump dataclass.
  - PK quarter_cad (natural key, one row per quarter)
  - quarter_geom (MultiPolygon, 4326) + bbox_3857 (Polygon, 3857) — dual-index
    стратегия: bbox для scheduler queries без transform, geom для joins с
    PZZ/district в 4326
  - 7 per-layer count columns + total_features (computed denorm для быстрых
    статистических queries без распаковки JSON)
  - features_json JSONB — array of ~117 NSPDFeature objects (~120KB per row,
    TOAST handles transparently). Geometry в raw EPSG:3857 — caller transforms.
  - layers_fetched TEXT[] mirrors QuarterDump.layers_fetched tuple
  - harvest_duration_ms (nullable — NULL = "not measured"), harvest_error TEXT
  - region_code SMALLINT NOT NULL (no DEFAULT — explicit per insert)
- 6 indexes: PK, GIST(bbox_3857), GIST(quarter_geom), B-tree(fetched_at_utc DESC),
  B-tree(region_code), partial B-tree(fetched_at_utc WHERE harvest_error IS NULL)
- View v_quarter_dumps_freshness — admin monitoring (AGE + is_failed flag)

Конвенции:
- BEGIN/COMMIT atomic
- IF NOT EXISTS на каждом CREATE — idempotent
- COMMENT ON TABLE + 7 COMMENT ON COLUMN + comments на каждом index
- psycopg v3 compatible (DDL only, no %s)

Применено к prod в рамках database-expert verification (DDL only, idempotent).

Vault entry: code/schemas/Schema_Nspd_Quarter_Dumps.md (upsert template +
EXPLAIN plans + cross-refs).

Code review (code-reviewer pre-push): APPROVE, 2 minor non-blocking:
- MINOR-1 quarter_geom MultiPolygon vs Polygon — resolved в Celery PR via
  ST_Multi(ST_Transform(...)) на ingest contract
- MINOR-2 layers_fetched default '{}' — documented в comment как
  partial-failure path для error rows

Next: Celery harvest_quarter task + beat schedule.

Part of #94 Sprint 1.1.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 18:22:50 +03:00
lekss361
9ee0b07003
feat(scrapers): search_by_quarter orchestrator + QuarterDump (#94 pt.2/4) (#109)
Sprint 1.1 item #1 из плана #94 part 2. Foundation для PKK harvest pipeline —
1 vacuum (search) + N layer fetches → comprehensive snapshot всех NSPD данных
в пределах квартала. Базис для G1 #28 ПЗЗ, G3 #30 ЗОУИТ, P2 #46 neighbors,
E1 #51 parcels backfill, #96 ЕГРН помещения.

Backend (nspd_client.py):
- New QuarterDump frozen dataclass (slots=True): quarter + per-layer feature
  lists (parcels/buildings/territorial_zones/red_lines/engineering + zouit
  dict + risks dict) + bbox_3857 + layers_fetched (immutable tuple) +
  fetched_at_utc + total_features property.
- New NSPDClient.search_by_quarter(quarter_cad, include_zouit=True,
  include_risks=False): search → bbox → bulk fetch per layer phase.
  Cost 6/11/22 requests.
- New _geojson_bbox_3857() module-level helper — recursive coord walker.
- Class constants QUARTER_CORE_LAYERS / QUARTER_ZOUIT_LAYERS /
  QUARTER_RISK_LAYERS.

Empty-quarter (NSPD не нашёл cad): quarter=None, bbox=None, all lists empty,
zouit/risks dicts populated с пустыми lists (структурно стабильно),
layers_fetched=('search',).

Tests: +12 tests (31 total, no network).

Code review (code-reviewer pre-push): MINOR, fixed 3 of 5:
- datetime import → module-level
- layers_fetched → tuple[str, ...] (immutable in frozen dataclass)
- docstring clarified empty-quarter semantics
Bonus: ruff UP038 isinstance tuple → union syntax.

Part of #94. Sprint 1.1: 4 PRs total. Next: migration → Celery → integration.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 18:11:30 +03:00
lekss361
bc092b58f1
feat(scrapers): NSPD client foundation (#94) — search + WMS + layers (#98)
* feat(scrapers): NSPD client foundation (#94) — search_by_cad + WMS + layers

Foundation для G1 #28 ПЗЗ, G2 #29 ВРИ, G3 #30 ЗОУИТ, E1 #51 cad_parcels,
I3 #44 engineering, и поддержки on-demand #93.

Backend (`app/services/scrapers/nspd_client.py`):
- `NSPDClient` с 4 core методами:
  - `search_by_cad(cad, thematic_id)` — REST /api/geoportal/v2/search
    возвращает GeoJSON + ВРИ + land_category + cost_value за 1 запрос
  - `get_feature_info(layer_id, lon, lat, buffer_m=100)` — WMS GetFeatureInfo
    на точке: какие feature'ы layer'а покрывают (lon,lat)
  - `get_features_in_bbox(layer_id, bbox_3857)` — bulk через WMS workaround
    (WFS GetCapabilities → 404, поэтому большой bbox + центральная I/J точка)
  - `list_layers(theme_id)` — каталог слоёв в теме (PKK=1, ARN=665)
- Typed responses: NSPDFeature, NSPDSearchResult, NSPDLayer (frozen dataclasses)
- LAYERS catalog: 32 layer-id с семантическими именами (territorial_zones=875838,
  zouit_engineering=37578, risk_flooding=872205, etc) — TIER 1-6 per #94
- Coordinate helpers: lonlat_to_3857(), bbox_around_point_m()
- Reuse: HEADERS + SSL ctx + fetch_geoportal из existing nspd_lite (WAF-compatible
  urllib trick). Rate limit через rate_ms.
- WAF/Rate-limit: raises NspdLiteWafError на 403/429 — caller backoff.

Tests (`backend/tests/test_nspd_client.py`): 14 unit tests, no network.
- LAYERS catalog sanity (territorial_zones=875838 закрывает G1)
- Coordinate transforms (zero, ЕКБ center, bbox)
- NSPDFeature.from_raw parsing (full / missing fields)
- NSPDSearchResult helpers (empty/first)
- _walk_layer_tree (flat / nested / empty)
- search_by_cad через monkeypatch fetch_geoportal

Также включён micro-fix из PR #95 review (cosmetic):
- parcels.py inline comment "до 25s" → "до 15s" (matched _INLINE_FETCH_WAIT_S)

Scope NOT в этом PR (отдельные follow-up issues, sequential rule):
- Schema migrations (cad_parcels_geom +columns, nspd_territorial_zones,
  nspd_zouit, nspd_red_lines, nspd_risk_zones tables) → отдельный PR
- Celery tasks (sync_territorial_zones_bbox, sync_zouit_*, sync_risk_*) →
  отдельный PR использует client
- Admin endpoints (trigger / status) → отдельный PR
- Замена on-demand fetch на nspd_client → PR after schema ready

Closes part of #94 (foundation only — sub-issues for schema/tasks).

* fix(scrapers): address PR #98 auto-review minor feedback

Backend (nspd_client.py):
1. (line 100+) bbox_around_point_m signature split на multi-line.
2. (lazy imports) import json, time → module-level.
3. (duplicate SSL ctx) Reuse _SSL_CTX из nspd_lite через explicit import.
4. (shape defense) list_layers чекает {"data": [...]} wrapper + non-dict/list garbage → warning + empty.
5. (cad_num docstring) search_by_cad документирует формат + ссылку на validate_cad_format.

Tests: +5 mock тестов (19/19 passed):
- get_feature_info URL builder + parsing
- get_features_in_bbox bbox propagation
- list_layers_walks_tree_response
- list_layers_handles_data_wrapper
- list_layers_handles_garbage_response

Backend (parcels.py): docstring '~25с' → '~15с'.

Per auto-review on be14388.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 09:47:45 +03:00
lekss361
49fac806fc
feat(site-finder): auto-fetch cadastre geometry on-demand (#93) (#95)
* feat(site-finder): auto-fetch cadastre geometry on-demand (#93)

Когда пользователь вводит cad-номер которого нет в БД (cad_quarters_geom /
cad_buildings / cad_parcels_geom), вместо 404 «Загрузи через NSPD geo» (dead-
end для non-admin) — теперь backend автоматически инициирует NSPD fetch,
frontend показывает loading state с polling /fetch-status каждые 2с.

Backend:
- Новый модуль app/services/site_finder/cadastre_fetch.py с helpers:
  find_or_enqueue_fetch (atomic check + enqueue с дедупликацией по cad),
  fetch_status (smart polling endpoint — отличает not_in_nspd от failed),
  detect_thematic_id (3-сегм quarter / 4-сегм parcel / 5-сегм building),
  validate_cad_format.
- Reuse: enqueue_geo_job + process_nspd_geo_job (workers/tasks/nspd_geo).
  source_kind='auto_on_demand' отличает от bulk; rate_ms=200, priority=9.
- POST /parcels/{cad}/analyze graceful fallback: inline await до 25s
  (fast path), затем 202 + job_id + eta_seconds для polling, либо
  400/404/503 в зависимости от статуса (с Retry-After 60s на 503).
- GET /parcels/{cad}/fetch-status новый endpoint для polling.

Frontend:
- useSiteAnalysis расширен: fetchingState + cancel(). POST analyze + при 202
  polling каждые 2с (max 60 итераций = 2 мин cap). status=ready → re-trigger
  analyze; not_in_nspd/failed/invalid_format → typed errors.
- apiFetchWithStatus<T> + HTTPError в lib/api.ts — status-aware variant
  для 200 vs 202.
- FetchingState.tsx: animated spinner, progress bar (linear до etaSeconds,
  asymptote после), elapsed counter, cancel button. Светло-голубая
  scheme отличается от обычного pending skeleton.
- site-finder/page.tsx: FetchingState когда fetchingState активен; обычный
  pending skeleton — только при первичном analyze без 202.

Edge cases (per #93 acceptance):
- cad валидный в НСПД, fetch <25s → inline 200 OK
- cad валидный, fetch >25s → 202 + frontend polls → ready → analyze
- cad валидный, не в НСПД → 404 с понятным сообщением + формат hint
- cad invalid format → 400 + формат hint
- NSPD rate-limited / failed → 503 + Retry-After 60s
- Параллельные запросы на тот же cad → один job, оба клиента poll'ят (дедуп
  через find_active_on_demand_job).

Closes #93.

* fix(site-finder): address PR #95 auto-review minor feedback

Backend (cadastre_fetch.py):
1. (race condition) Advisory lock `pg_try_advisory_xact_lock(hashtext(cad_num))`
   обёрнут вокруг шагов "check active job → enqueue" в find_or_enqueue_fetch.
   Lock transaction-scoped, released at COMMIT. Параллельные запросы на тот
   же cad: первый получает lock и enqueue; второй lock=false → re-check
   active job (уже виден после первого COMMIT) → возвращает тот же job_id.
   Docstring обновлён, упоминание SELECT FOR UPDATE удалено.

Backend (parcels.py):
3. (threadpool exhaustion) _INLINE_FETCH_WAIT_S снижен 25 → 15s с подробным
   комментарием: tradeoff про default Starlette anyio threadpool (40 slots)
   и concurrent burst saturation. 15s баланс: НСПД avg 5-15s для quarter,
   ~70% fast path; остальные 30% получают 202 + polling без блока.

Data (87_on_demand_indexes.sql):
2. (missing index) New migration:
   - `nspd_geo_targets_cad_num_idx ON nspd_geo_targets(cad_num)` — для
     find_active/recent_completed_job (existing UNIQUE composite не покрывает
     WHERE cad_num=:c).
   - `nspd_geo_jobs_source_status_idx ON nspd_geo_jobs(source_kind, status)`
     composite для filter auto_on_demand + queued/running.
   Idempotent (CREATE INDEX IF NOT EXISTS), не блокер при текущем размере,
   критично при росте on-demand traffic.

Frontend (useSiteAnalysis.ts):
4. (UI flicker) setFetchingState(null) перемещён ПОСЛЕ `await second
   apiFetch`. Иначе между clear и resolve есть момент когда mutation
   isPending=true + fetchingState=null → пустой экран ~1 RTT.

NOT addressed (rebuttal):
5. (Tailwind convention) — проверил: в проекте нет ни globals.css, ни
   @tailwind directives. ВСЕ существующие site-finder components используют
   inline styles (ConfidenceBadge, GeotechRiskBlock, ScoreBreakdownPanel etc).
   Tailwind в deps но не wired up. Keep inline styles для consistency.
6. (animate-spin) — требует Tailwind globals (см. #5). `<style jsx>`
   keyframes — built-in Next.js, работает.

Per auto-review on 2252236.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 09:02:17 +03:00
lekss361
f4a865060f
feat(site-finder): D4 pipeline 24mo — future competition (#36) (#90)
* feat(site-finder): D4 pipeline 24mo — future competition (#36)

Backend (parcels.py):
- Запрос к domrf_kn_objects в радиусе 5км с ready_dt BETWEEN NOW() AND NOW()+24mo.
- _aggregate_pipeline() — сводка: objects_count, flats_total, by_class (эконом/
  комфорт/бизнес/...), by_quarter (хронологически, для UI bar), severity
  (low <500 / medium <3000 / high) per spec, top_objects (десятка по flat_count desc).
- Поле analyze.pipeline_24mo. Backward-compat — optional.

Frontend:
- Pipeline24moBlock.tsx — severity badge + 3 summary numbers (объектов, квартир,
  горизонт/радиус), by-class chips, гистограмма bar по кварталам сдачи
  (нормирована на max), разворачиваемый top-N список с классом + датой сдачи.
- Добавлен в MarketTab выше "Market trend".
- TS типы: Pipeline24mo, PipelineObject, PipelineQuarterSlot.

Closes #36. Relates to #19 (Конкурентный 360 — закрыт ранее в #36 scope).

* fix(site-finder): address PR #90 auto-review feedback

Must-fix (3):
1. distance_m falsy guard: `if obj.get("distance_m") is not None` вместо
   `if obj.get("distance_m")` — centroid-on-building даёт 0.0 (falsy float),
   raw Decimal иначе упал бы в JSON serialization.
2. SQL plan note добавлен про seq scan ~3000 строк OK; при росте — нужен
   GIST/index на (latitude, longitude) — отдельный issue для database-expert
   (будет создан separately).
3. obj_class NULL bug помечен в docstring _aggregate_pipeline с reference на
   fixes/Bug_Kn_API_Obj_Class_Always_Null_OPEN. D6/#38 — fix плановый.

Cleanup (3 из 5):
4. CLASS_LABEL.null dead key убран — JSON null приходит as absent key, не
   "null" string.
6. Magic numbers вынесены: PIPELINE_RADIUS_M=5000, PIPELINE_HORIZON_MONTHS=24,
   PIPELINE_SEVERITY_MEDIUM_THRESHOLD=500, PIPELINE_SEVERITY_HIGH_THRESHOLD=3000,
   PIPELINE_TOP_OBJECTS_LIMIT=10. SQL query теперь через f-string подставляет
   их (защищённое от injection — это int литералы).
8. obj.ready_dt formatting через fmtMonth() с new Date + toLocaleDateString —
   robust к datetime suffix vs date-only, fallback к substring(0,7) при NaN.

Не сделано (defer):
5. Async 3 HTTP calls (pre-existing pattern, нужен ThreadPoolExecutor refactor
   отдельным PR — затрагивает weather/air_quality fetch architecture).
7. ST_GeomFromText дважды — CSE справляется на этом масштабе.

Per auto-review on ade511b.

* fix(site-finder): address PR #90 auto-review minor feedback

1. TS PipelineObject.distance_m — `number | null` для отражения defensive
   Python guard (`if obj.get("distance_m") is not None`). Comment объясняет
   почему.
2. Pipeline SQL: `text(f"...")` → `text("...")` + parameters. radius_m и
   horizon_months через `:param` placeholders + `cast(:horizon_months || ' months'
   AS interval)`. Consistency с остальными SQL в файле, plus защита от
   accidental injection при будущих изменениях.
3. top_objects: explicit field selection вместо `dict(r) for r in rows`.
   Раньше leak'ило все колонки из CTE `SELECT *` (latitude/longitude/
   snapshot_date/region_cd/dev_id) в API response. Теперь только nominated
   fields: obj_id, comm_name, dev_name, obj_class, flat_count, ready_dt,
   distance_m. Schema clean.

Per auto-review on 4e431bf.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 08:34:02 +03:00
lekss361
1f8fd77825
feat(site-finder): X1 score breakdown + verbal explain (#47) (#92)
Backend (parcels.py):
- POI scoring loop teper строит score_breakdown_detailed: per-factor list с
  verbal explain (через _verbal_for_poi helper) и группировкой
  (_POI_GROUP: Социалка / Торговля / Парки / Транспорт / Шум/трамвай / Локация).
- center_bonus добавлен как synthetic factor группы "Локация" с weight=1.0
  (decay не применяется — bonus IS the value).
- factor key включает enumerate idx — prevents React key collision когда
  два POI одной категории совпадают по округлённому расстоянию.
- Skip факторов с |contribution| < 0.01 (POI > 1км) — UI шуму не нужен.
- abs_total fallback на 1.0 — защита от division-by-zero для empty factors.
- Top-3 positives/negatives: explicit ascending sort для негативов
  ("most-negative first" очевидно из кода).
- score_by_group: stacked-bar данные с count + contribution_pct.
- group_totals type: dict[str, dict[str, float | int]] (count это int).

Frontend:
- Новый ScoreBreakdownPanel.tsx: stacked-bar по группам с tooltip + legend,
  топ-3 плюса (▲ зелёный) / топ-3 минуса (▼ красный) с verbal, отдельная
  строка "Снижают балл — Шум/трамвай: ..." для negative groups, разворачиваемая
  таблица всех факторов (sticky thead, scrollable).
- Интегрирован в OverviewTab под секцией "Балл".
- TS типы: FactorContribution, ScoreGroupTotal.

Closes #47.

NB: branch создан заново из-за rebase mess (см. PR #87 comments). Логически
эквивалентно но history clean.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 08:17:00 +03:00
lekss361
09611dc690
feat(site-finder): P1 geometry suitability score (#45) (#89)
* feat(site-finder): P1 geometry suitability score (#45)

Backend (parcels.py):
- _polygon_suitability() — Shapely metrics on parcel WGS84 polygon:
  area_ha/m2 (projected via cos(lat) к UTM-like meters), perimeter_m,
  aspect_ratio (long/short side of MABR), convex_hull_ratio (area / hull area),
  min_inscribed_rect_dim_m.
- Composite suitability score 0..1:
  base = area_subscore (0 при <0.2 ha, linear → 1 при ≥0.5 ha)
  −0.3 если aspect_ratio > 5 (вытянутый)
  −0.3 если convex_hull_ratio < 0.65 (изрезанный)
  −0.5 если min_inscribed_rect_dim_m < 30 (узкий)
- Label: микро / подходящий / сложная форма / слабо подходит
- Recommendation: "для МКД 16+ нужно >0.3 ha + минимум 40м"
- В analyze response → geometry_suitability.

Frontend:
- Новый GeometrySuitabilityBlock.tsx с color-coded badge + метрики grid +
  penalties + recommendation
- Добавлен в LandTab (выше "Зонирование (ПЗЗ)")
- TS типы расширены: GeometrySuitability

Closes #45.

* fix(site-finder): address PR #89 auto-review minor feedback

Backend (parcels.py):
1. Фактическая ошибка фикс: recommendation теперь различает "строительный
   минимум 30м" (physical penalty trigger) и "комфорт МКД 12-16эт 40м"
   (recommendation level). Пользователю чётко видны 2 порога.
2. Lazy imports → module-level: `from shapely import wkt as _shp_wkt`,
   `from shapely.geometry import Polygon`.
3. MABR inner exception теперь логирует: `logger.debug("MABR computation
   failed, falling back to sqrt(area): %s", mabr_err)`.
4. Magic numbers вынесены в константы: _GEOM_MIN_AREA_HA, _GEOM_AREA_SCORE_FULL_HA,
   _GEOM_ASPECT_PENALTY_THRESHOLD, _GEOM_CONVEX_PENALTY_THRESHOLD,
   _GEOM_MIN_WIDTH_PHYSICAL_M, _GEOM_MIN_WIDTH_COMFORT_M, _GEOM_LABEL_MICRO_HA,
   _GEOM_LABEL_GOOD, _GEOM_LABEL_MEDIUM, плюс penalty константы.
5. Label "микро" теперь комбинируется с penalties: "микро, узкий" — UI видит
   обе проблемы. Pure "микро" остаётся когда нет penalty.

Frontend (GeometrySuitabilityBlock.tsx + types):
6. Заменил emoji ⚠ (U+26A0) на текстовый <strong>"Проблемы формы:"</strong> —
   стабильнее в WeasyPrint PDF-экспорте и cross-platform.
- TS типы: GeometrySuitabilityBaseLabel экспортирован, label расширен до
  `string` для допуска combo-labels; helper colorForLabel() парсит base часть.

Per auto-review on f4e7491.

* fix(site-finder): P1 #45 — extract 0.3 ha to _GEOM_AREA_COMFORT_HA

Auto-review нашёл: recommendation упоминала "от 0.3 га" но в constants block
не было 0.3 — magic literal в строке. Тот же класс проблемы что 30/40м inconsistency
из прошлого review, только в другом поле.

Fix: новая константа `_GEOM_AREA_COMFORT_HA = 0.3` с комментарием
"рекомендуемая комфортная площадь МКД (recommendation)". Размещена между
_GEOM_MIN_AREA_HA (физический минимум) и _GEOM_AREA_SCORE_FULL_HA (premium) —
третий semantic threshold. Recommendation теперь f-string использует константу.

Per auto-review on ae5e8de.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 08:00:12 +03:00
lekss361
843314c040
feat(site-finder): P2 cad_buildings соседи + overlap check (#46) (#91)
* feat(site-finder): P2 cad_buildings соседи + overlap check (#46)

Backend (parcels.py):
- _parse_floors() helper для TEXT column (cad_buildings.floors хранится как
  строка, могут быть диапазоны "5-7"). Возвращает верхнюю границу.
- _neighbors_summary(db, geom_wkt, our_cad) — query соседей в 100м (GIST):
  cad_num, building_name, floors, year_built, cost_value, area, address, distance.
  Aggregate: avg_floors_100m, max_floors_100m, median_cost_per_m2_100m,
  count_buildings_100m. Outliers cost/m² фильтруются (1k < x < 500k).
- Overlap check: ST_Intersects + ST_Area(ST_Intersection) > 50 m² (transformed
  to UTM 32641 для метров). Если есть → has_existing_buildings: true +
  overlap_buildings list.
- В response → neighbors_summary.

Frontend:
- Новый NeighborsBlock.tsx: hard red warn block для overlap (с building names +
  overlap_m2 + "Инвестиции невозможны без сноса"); summary metrics (avg/max
  floors, median price); toggle "Показать N ближайших" → таблица.
- Border меняется на красный при has_existing_buildings — visual cue.
- Добавлен в LandTab выше "Зонирование (ПЗЗ)".
- TS типы: NeighborBuilding, OverlapBuilding, NeighborsSummary.

Closes #46. Closes #21 (cad_buildings в Site Finder фильтрах).

* fix(site-finder): address PR #91 auto-review minor feedback

Backend (parcels.py):
1. (medium) Aggregation loop _neighbors_summary теперь обёрнут в try/except
   (ValueError, TypeError) с fallback к data_available:False + log warning.
   Защищает от non-numeric cost_value/area придёт в строке (e.g. "N/A") —
   ранее весь endpoint падал 500.
2. Magic numbers вынесены: _COST_PER_M2_MIN=1000, _COST_PER_M2_MAX=500_000.
3. _parse_floors docstring + inline note про malformed parts ("5а-7" filter,
   multi-range "1-2-3" max acceptable degradation).

Frontend (NeighborsBlock.tsx):
5. Русский plural fix: pluralBuildings(n) helper — 1 здание, 2-4 здания,
   5+/11-14 зданий. Раньше "3 зданий" — теперь "3 здания".

Не сделано (defer):
4. ST_Area для overlap query — практически 0-5 buildings в ЕКБ, GIST scan OK.
6. Discriminated union для NeighborsSummary — refactor а не bug.
7. Vault entry для P2 — добавится batch'ем после merge всех текущих PR.

Per auto-review on 60d53bb.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 01:30:53 +03:00
lekss361
3777a77a42
feat(site-finder): X2 confidence indicator + caveats (#48) (#88)
Backend (parcels.py):
- _compute_confidence() composite score 0..1 from 7 subscores: poi_freshness,
  geom_source (parcel vs quarter), district, market_trend (rosreestr_deals depth),
  competitors, environment (noise/air/weather availability), zoning (placeholder
  до G1).
- confidence_label: high (>0.75) / medium (0.4-0.75) / low (<0.4)
- confidence_caveats: list of конкретных проблем для UI
- confidence_breakdown: per-subscore 0..1 для прозрачности

Это stub-версия (полная — после G1/G2/D1/D2). Использует только текущие сигналы.

Frontend:
- Новый ConfidenceBadge.tsx — color-coded (green/yellow/red) badge с %
- Caveats для low — показываются сразу; для medium/high — под toggle
- Toggle "Подробнее" → breakdown per-subscore + полный список caveats
- Размещён в начале OverviewTab (выше "Район")
- TS типы расширены: confidence, confidence_label, confidence_breakdown, confidence_caveats

Closes #48.

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 01:04:25 +03:00
lekss361
485c221c3b
docs(claude): formalize Auto-review passed as approval signal (#85)
* docs(claude): formalize "Auto-review passed" as approval signal + polling loop

Per user feedback after PR #84 merge:
- Critical rule 5: "Auto-review passed" / "LGTM" от auto-review bot = approval
  (наряду с явным "merge it" / "approved" от human user)
- Add "Polling loop для PR" section: 60s ScheduleWakeup pattern,
  decisive states (merged / approved / changes / no-news), cap 30 iters

This formalizes what main session already did: after fixup commit, poll
comments, detect auto-review verdict, and auto-merge if passed.

* fix(claude): address PR #85 auto-review feedback

- Fix cyrillic/latin typo (rule 5 second sub-bullet)
- Add SHA marker validation to polling loop (race condition on stale LGTM after fixup-push)
- Add explicit "Auto-merge scope" rule 6 — allowlist (docs/UI/.claude) vs blocklist (migrations/API/infra/secrets)
- Sync "Когда auto-mode" section with new merge approval signals
- Lower cap to 5 iters for scope-blocked PRs (human required anyway)
- Self-modifying CLAUDE.md rules → always require human

Per auto-review on e66e1ea.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 00:43:25 +03:00
lekss361
a35a3d02ea
feat(site-finder): deep-link tabs via ?tab= URL param (#86)
* feat(site-finder): deep-link tabs via ?tab=env|land|market URL param

Tab state now syncs with URL — можно поделиться ссылкой на конкретный
таб (например /site-finder?tab=market для тренда цен по последнему cad).

- useRouter + useSearchParams (Next 15 app router, client-component)
- router.replace (не push) чтобы tab switches не засоряли history
- useEffect on searchParams → back/forward работают
- overview = default, не пишется в URL (clean)
- Type-guard isTabId(): только valid values из URL переключают state

* fix(site-finder): address PR #86 auto-review minor recommendations

- Use usePathname() for router.replace fallback (avoid dangling "?" in Next 15)
- Remove stale-closure guard in useEffect — React bails Object.is, no eslint-disable
- Wrap page in <Suspense> for useSearchParams (Next 15 App Router req for static)
- Lazy useState initializer — skip wasted CPU on re-renders

Per auto-review on c47e56b.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 00:33:49 +03:00
lekss361
aadf47709b
docs(claude): mandate branch+PR workflow, ban direct push to main (#84)
* docs(claude): mandate branch+PR workflow, ban direct push to main

* docs(claude): sync TL;DR + Don't-do-these with new PR workflow

Address auto-review feedback:
- TL;DR rule 5 → branch+PR mandate (was: "user commits themselves")
- TL;DR rule 6 → no --no-verify/--force/--amend (split from rule 5)
- Don't-do-these → drop conflicting "no auto-commit" line, add
  "no direct push to main" + "no merge without approval"
- Critical workflow rules: add explicit rule 2 — worker-agents don't
  commit themselves; main session commits on feature branch
- Restore "Squash on merge" wording in PR section (was dropped)
- Auto-mode section: clarify that main session does commits on feature
  branch (worker-agents leave changes staged)

* fix(claude): Code-reviewer diagram had stale 'push origin HEAD:main' — replace with branch+PR flow

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-12 00:16:06 +03:00