Commit graph

19 commits

Author SHA1 Message Date
bd1ed4861c docs(zouit): уточнить DISTINCT ON-комментарий + убрать stale cad_zouit из FE-типа (#1957)
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (pull_request) Successful in 1m0s
CI / openapi-codegen-check (pull_request) Successful in 1m51s
CI / backend-tests (pull_request) Successful in 12m4s
Code-review nits (без изменения поведения):
- quarter_dump_lookup.py: комментарий про дедуп ошибочно утверждал, что зоны без
  рег-номера не схлопываются. На деле reg_numb_border — TEXT NOT NULL (0 NULL на
  проде), а DISTINCT ON в PG считает NULL равными. Переформулировано на точное.
- frontend/types/nspd.ts: убран устаревший 'cad_zouit' из списка возможных
  group_key (бэкенд теперь эмитит только protected/engineering/okn/natural/other).
2026-06-27 14:21:01 +05:00
76647bb4cf fix(report): площадь/обновлено EGRN, forecast-confidence dedup, ЗОУИТ типизация (#1953)
#1954 — площадь «—»: COALESCE(land_record_area, specified_area, declared_area)
в EGRN-блоке analyze (land_record_area NULL у 12809/42233 участков, но
specified_area заполнена). area_m2 для 66:41:0205010:287 теперь 106378 (был NULL).

#1954 — «Обновлено» сломано: cost_registration_date — мёртвая колонка (0/42233);
репойнт на updated_at (42233/42233 заполнено). Ключ ответа last_egrn_update_date
не меняется (additive value fix).

#1958 — confidence-фактор «Прогноз спрос/предложение» дублировался ×4
(по фактору на горизонт). Сворачиваем в один weakest-link'ом (MIN ранга
по горизонтам) в _component_confidences до confidence-движка (#990).

#1957 — ЗОУИТ backend:
- _get_cad_zouit_overlaps: DISTINCT ON (reg_numb_border) — дедуп дубль-строк
  (одна физ. зона 2× с разным category_name). group_key cad_zouit→protected.
- _get_zouit_overlaps (dump path): subcategory→RU-тип карта (26→СЗЗ и др.,
  коды сверены кросс-джойном dump↔cad_zouit на проде); type_zone из карты,
  reg из props.options.reg_numb_border. Раньше отдавал blank-строки.
- унификация group_key (protected/engineering/okn/natural/other) + top-level
  reg_numb_border в обоих путях.

UP038-модернизация isinstance в report_assembler (pre-commit ruff 0.7.4).

Frontend note (#1957): nspd_zouit_overlaps теперь всегда group_key из набора
{protected,engineering,okn,natural,other} — сырой 'cad_zouit' больше не отдаётся;
оба пути несут type_zone + reg_numb_border.

Tests: +4 _component_confidences collapse, +6 ЗОУИТ (subcategory map, dump
typing, DISTINCT ON), schema-test обновлён на protected. 451 passed targeted.
2026-06-27 14:14:40 +05:00
bc1da8eac9 fix(gate): area-gate ЗОУИТ utility-easement blocker (sub-17) — снять over-block
All checks were successful
CI / changes (pull_request) Successful in 7s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Successful in 1m52s
CI / backend-tests (pull_request) Successful in 11m34s
ZOUIT_OVERLAP_SUB17 (единственный subcategory-blocker = охранная зона
ЛЭП/газа/трубопровода) был ХАРД-блокером на ЛЮБОЕ пересечение → блокировал
434/574 (75.6%) участков как «нельзя строить МКД». Прод-замер: реальное
покрытие участка этими зонами median=6%, p90=47%, только 1/17 >60% — тонкие
инж-коридоры. По РФ охранная зона ограничивает застройку ВНУТРИ полосы, а не
весь участок (МКД сажается на незанятой части). Хард-блок на 6%-покрытии неверен
и (после фикса резолва зон #1891) прятал financial_estimate у ~90% участков.

Фикс: area-gate.
- quarter_dump_lookup._get_zouit_overlaps + _get_cad_zouit_overlaps: добавлен
  coverage_pct (доля площади участка под зоной, в geography м², NULLIF-guard).
- gate_verdict.compute_gate_verdict: sub-17 (и cad network/blocker) АГРЕГИРУЮТСЯ
  (sum-capped, консервативно к блоку) → блокер ТОЛЬКО если покрытие >
  settings.gate_zouit_engineering_blocker_min_coverage (0.6, env-tunable), иначе
  WARNING ZOUIT_ENGINEERING_PARTIAL с % покрытия. Больше нет 4× дубль-блокеров.
  СЗЗ/OKN/прочие ЗОУИТ — без изменений (warnings). main_vri/резидентность — без
  изменений.
- coverage_pct отсутствует (legacy/synthetic) → трактуем как 0 → warning
  (документировано: дамп всегда даёт coverage; ложный хард-блок дороже).

Тесты: +помесь gate/aggregation/coverage (131 combined passed); правка
_make_zouit_row (7-я колонка coverage_pct) в tests/test_quarter_dump_lookup.py.
Полный бэкенд локально: 2 failed → fixed → перепроверка. ruff чисто. Схема не
менялась (gate_verdict/overlaps — dict[str,Any], api-types regen не нужен).

Refs #1881
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 14:45:59 +05:00
cbe509f092 fix(site_finder): read real NSPD territorial-zone keys in _get_zoning
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Successful in 1m51s
CI / backend-tests (pull_request) Successful in 8m52s
_get_zoning() read non-existent props (top-level reg_numb_border / zone_code /
type_zone / name) so nspd_zoning.zone_code & zone_name were NULL on every parcel.
Real dump props (verified prod, 796 features) carry only the registration number,
duplicated across descr / label / externalKey / options.reg_numb_border (no human
zone name exists in the NSPD dump). Map both fields to that reg-number with safe
nested narrowing (isinstance options dict) + legacy fallbacks; honest — no
fabricated name. Surfacing reg_numb_border also unblocks the КСИТ crosswalk (#1843).
Regression tests use the real prop shape (full / nested-only / opaque→None).
2026-06-20 20:29:45 +05:00
479f01ae1a Merge remote-tracking branch 'forgejo/main' into HEAD
Some checks failed
CI / changes (push) Successful in 7s
CI / changes (pull_request) Successful in 8s
CI / frontend-tests (push) Successful in 1m7s
CI / openapi-codegen-check (push) Failing after 2m9s
CI / frontend-tests (pull_request) Successful in 1m4s
CI / openapi-codegen-check (pull_request) Failing after 1m37s
CI / backend-tests (pull_request) Failing after 9m7s
CI / backend-tests (push) Failing after 9m11s
# Conflicts:
#	backend/app/services/site_finder/gate_verdict.py
2026-06-15 21:07:05 +03:00
86e9ea2937 fix(week-review): автофиксы код-ревью — 169 issue (label «week ревью 1»)
Многоагентный аудит + имплементация: один воркер на файл, точечные правки.
Верификация: py_compile (47/47 .py) + tsc --noEmit (0 ошибок). Unit-тесты
не прогонялись (окружение не поднято: rollup native dep / нет pytest-venv).

Полностью исправлено (169): #1336, #1337, #1339, #1340, #1341, #1342, #1343, #1345, #1346, #1348, #1349, #1350, #1351, #1354, #1356, #1358, #1359, #1360, #1362, #1364, #1365, #1366, #1367, #1368, #1369, #1370, #1371, #1372, #1373, #1374, #1375, #1376, #1377, #1378, #1379, #1380, #1381, #1382, #1384, #1385, #1386, #1387, #1388, #1389, #1390, #1391, #1392, #1394, #1395, #1396, #1397, #1399, #1400, #1401, #1402, #1403, #1404, #1408, #1409, #1410, #1411, #1412, #1413, #1414, #1415, #1416, #1417, #1418, #1420, #1423, #1425, #1426, #1427, #1428, #1429, #1430, #1431, #1432, #1433, #1434, #1435, #1437, #1438, #1439, #1440, #1441, #1442, #1443, #1444, #1445, #1446, #1447, #1448, #1449, #1450, #1451, #1452, #1453, #1454, #1455, #1456, #1457, #1458, #1459, #1460, #1461, #1462, #1463, #1464, #1465, #1466, #1467, #1468, #1469, #1471, #1472, #1473, #1474, #1476, #1478, #1479, #1481, #1482, #1483, #1484, #1485, #1487, #1488, #1489, #1490, #1491, #1492, #1493, #1494, #1495, #1496, #1497, #1499, #1500, #1501, #1502, #1504, #1505, #1506, #1507, #1510, #1514, #1515, #1516, #1517, #1518, #1519, #1521, #1522, #1523, #1524, #1525, #1526, #1527, #1528, #1529, #1531, #1532, #1533, #1534, #1535, #1536, #1537, #1538

Частично (9, in-file часть, остаток cross-file): #1361, #1419, #1422, #1424, #1470, #1475, #1477, #1480, #1498
Требуют cross-file (3, не тронуты): #1338, #1363, #1421
Пропущено (1): #1539

Не входило в партию: 22 needs-Leha issue (нужны решения владельца).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 20:21:11 +05:00
e5e0ccb283 feat(site-finder): сетевые обременения из ЗОУИТ (#1070, Refs #103) (#1328)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m36s
Deploy / build-worker (push) Successful in 2m40s
Deploy / deploy (push) Successful in 1m12s
2026-06-14 14:49:32 +00:00
50555821fb feat(site_finder): routing POI decay (#41) + OSM utility distances (#44) + ЗОУИТ geom (#255bk)
Some checks failed
CI / changes (push) Successful in 7s
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 7s
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (push) Successful in 7m42s
CI / backend-tests (pull_request) Successful in 7m40s
Deploy / changes (push) Has been cancelled
Deploy / build-backend (push) Has been cancelled
Deploy / build-worker (push) Has been cancelled
Deploy / build-frontend (push) Has been cancelled
Deploy / deploy (push) Has been cancelled
#41: per-category routing-radius decay через reusable ORS /matrix client (foot-walk
время + per-category пороги + decay-curves), opt-in ?decay=routing, graceful straight-line
fallback (analyze не 500 при ORS down) + length-guard durations==destinations.
#44: +5 OSM-категорий (transformer/gas/water/sewerage/heat) via Overpass nwr;
nearest_{water_main,substation,gas,heat}_m в analyze.utilities.
#255bk: ST_AsGeoJSON(CAST(geom AS geometry)) в _get_cad_zouit_overlaps → geom_geojson.

Closes #41
Closes #44
2026-06-13 22:31:56 +05:00
2e428405d2 fix(site-finder): расширить except в parcel_ird_overlaps до DataError (#1095)
parcel_ird_overlaps ловил только (OperationalError, ProgrammingError).
При D9b-wiring в analyze malformed WKT в ST_GeomFromText давал PostGIS
ERROR → SQLAlchemy DataError, который пробивал try/except → analyze
падал вместо graceful-degrade.

- ird_overlay_lookup: + DataError в imports и в except tuple.
- quarter_dump_lookup: симметричное расширение в _get_engineering_*.
- test_ird_overlay_lookup: parametrized test_graceful_when_db_fails
  покрывает 3 класса (Operational/Programming + DataError для WKT).

Closes #1095
2026-06-13 15:02:50 +05:00
lekss361
81cf274e80 fix(#234): address re-review nits — Settings singleton + docstring sync
1. _acquire_harvest_lock: Settings() instance → singleton 'settings' import
   from app.core.config — соответствует convention (scrape_kn.py, objective.py,
   celery_app.py все используют singleton).
2. _trigger_harvest docstring: убрана incorrect фраза «остальные ... видят
   harvest_eta_seconds от первого запроса» (код возвращает eta=None для
   denied-lock callers). Заменена на actual behavior: graceful no-data UI
   с ПКК fallback на FE.

PR #250 re-review followup (issues #3 + docstring sync).
2026-05-17 09:31:43 +03:00
lekss361
b99ac76463 Merge remote-tracking branch 'forgejo/main' into fix/234-nspd-harvest-eta-badge
# Conflicts:
#	backend/app/services/site_finder/quarter_dump_lookup.py
2026-05-17 09:19:12 +03:00
lekss361
990c109756 fix(#234): NSPD harvest ETA badge + SETNX dedupe (Sub-PR C)
Backend (quarter_dump_lookup.py):
- _acquire_harvest_lock: Redis SETNX TTL=120s на quarter, защищает от burst
  N concurrent analyze, ставящих N одинаковых harvest task в очередь
- _trigger_harvest: использует lock перед apply_async, возвращает False если
  lock уже взят (другой запрос триггернул раньше)
- make_empty_result/EMPTY_DUMP_RESULT: новое поле harvest_eta_seconds в
  nspd_dump dict, типичный harvest_quarter = 60с
- /analyze: пробрасывает поле через nspd_dump dict (нет typed schema —
  response_model=None для /analyze endpoint, dict уходит как есть)

Frontend (NspdFreshnessBadge, NspdZoningBlock):
- Countdown «НСПД: загрузка ~Nс» вместо бесконечного спиннера
- После остановки countdown (remaining=0) NspdZoningBlock показывает
  «загрузка дольше обычного» + ссылку на ПКК вместо infinite skeleton

Tests: 5 новых unit + 2 для empty_result schema (всего +7, pass)

Closes #234 (UX-side; data-side resolves когда Sub-PR B + D merged).
2026-05-17 09:13:06 +03:00
10b79da2e9 fix(#243): cad_zouit fallback срабатывает когда nspd dump полностью отсутствует (#248)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / deploy (push) Successful in 46s
Deploy / build-backend (push) Successful in 1m36s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m44s
2026-05-17 06:03:19 +00:00
lekss361
f70be684da feat(#29,#232): wire cad_parcels.permitted_use in analyze + cad_zouit fallback (G2+G3)
TASK A (#29 G2): add parcel_meta to analyze response
- New ParcelMeta Pydantic schema in app/schemas/parcel.py
- SELECT from cad_parcels WHERE cad_num=:c in analyze_parcel() (step 9f)
- Returns permitted_use_established_by_document, land_record_category_type,
  land_record_subtype, cost_value; None when row absent
- Tests: test_analyze_parcel_meta.py (found + not-found cases)

TASK B (#232 G3): cad_zouit fallback in _get_zouit_overlaps
- When nspd_quarter_dumps has zouit_count==0, fall back to ST_Intersects
  query on cad_zouit (3483 rows, GIST indexed)
- Overlaps tagged with source='cad_zouit'; format compatible with NSPD path
- gate_verdict.py: BLOCKER_TYPE_ZONE_KEYWORDS tuple for keyword-based
  classification (охранная зона / трубопровод / электр / газ -> blocker;
  СЗЗ -> warning); NSPD subcategory path preserved backward-compat
- Tests: 6 new test cases in test_gate_verdict.py covering cad_zouit path
  and backward-compat for NSPD subcategory path

Updated db.execute call sequence in test_analyze_*.py (index shift +1 at pos 10).
2026-05-17 08:17:22 +03:00
lekss361
fdb54834e5 fix(nspd): rename migration 89→98, fix red lines ST_Area→ST_Length (#220)
- data/sql/98_*: rename from 89 to avoid collision with existing 89_drop_dead_brin
- _get_red_lines: ST_Length(ST_Intersection(planar)::geography) instead of
  ST_Area(ST_Intersection(::geography, ::geography)) — fixes PostGIS 3.4
  tolerance error and returns correct non-zero length for LINESTRING intersections
- Rename intersection_area_sqm → intersection_length_m across schema, TS types,
  frontend component, and tests; add test_get_red_lines_db_exception_returns_empty

Addresses review-bot feedback on PR #220.
2026-05-16 19:33:14 +03:00
lekss361
ed3c128528 feat(nspd): TIER 4 opportunity layers + red lines (#94 PR2 of 4)
- NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt)
  + QuarterDump.opportunity field
- nspd_sync: harvest_quarter accepts include_opportunity, denorm cols
  has_auction_parcels + opportunity_count in UPSERT
- quarter_dump_lookup:
  - _get_opportunity_parcels (sort by distance, early-exit on count=0)
  - _get_red_lines (query existing dump.red_lines core path, layer='red_lines')
- SQL 89_*: has_auction_parcels + opportunity_count + partial index
- Pydantic: OpportunityParcel + RedLine schemas
- /analyze: nspd_opportunity_parcels + nspd_red_lines fields
- Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration
- Tests: 28 total (11 PR1 + 17 PR2), all pass

Red lines uses existing core harvest path (layer 879243 already in dump.red_lines
from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed.

Part of #94
2026-05-16 19:06:22 +03:00
lekss361
1595157ebf feat(nspd): enable TIER 3 risk zones harvest + extract in analyze (#94 PR1)
- harvest_stale_quarters fanout passes include_risks=True to all queued tasks
- _get_risk_zones() with ST_Intersects + ST_Area intersection calc
- _extract_features_by_layer() generic helper for future PR2+ reuse
- RiskZone Pydantic schema; /analyze returns nspd_risk_zones list
- NspdRiskZonesBlock component with severity color-coding (red/yellow/orange)
- 11 unit tests for extract/intersect/null-area/db-exception paths

Part of #94 (TIER 3 risk layers — flooding, landslide, burns, erosion etc)
2026-05-16 16:11:59 +03:00
4deb1b0284 feat(parcels): connection-points endpoint (Forgejo #115 Phase 1 backend) (#190)
Some checks failed
Deploy / changes (push) Successful in 5s
Deploy / build-frontend (push) Failing after 0s
Deploy / build-worker (push) Successful in 5m11s
Deploy / deploy (push) Failing after 0s
Deploy / build-backend (push) Successful in 3m44s
2026-05-16 05:58:27 +00:00
lekss361
1c2f80a4b8
feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 FINAL) (#116)
* feat(site-finder): integrate nspd_quarter_dumps cache в analyze_parcel (#94 Sprint 1.1 #4 FINAL)

Замыкает Sprint 1.1 из #94 part 2 plan. После этого PR пользователь видит
свежие НСПД данные в UI (frontend integration — отдельный PR).

Backend (new app/services/site_finder/quarter_dump_lookup.py):
- `derive_quarter_cad(cad_num)` — 3/4/5-сегмент → quarter (3-segment)
- `get_quarter_dump_data(db, cad_num, parcel_wkt)` — main entrypoint:
  - Reads nspd_quarter_dumps row для derived quarter
  - Freshness threshold: 180 days
  - Missing/stale/harvest_error → trigger harvest_quarter.apply_async() fire-
    and-forget (lazy import против circular), return EMPTY_DUMP_RESULT
  - Fresh + parcel_wkt=None → metadata only (no spatial queries)
  - Fresh + geometry → 3 spatial queries via jsonb_array_elements + ST_Transform
    (3857→4326) + ST_Intersects / ST_DWithin
- 3 private helpers:
  - `_get_zoning` — point-in-polygon parcel centroid vs territorial_zones, LIMIT 1
  - `_get_zouit_overlaps` — все zouit_% layers пересекающиеся с parcel
  - `_get_engineering_nearby` — engineering_structures в 200m, sorted by distance
- `EMPTY_DUMP_RESULT` module-level constant — DRY для no-dump fallback (used
  in get_quarter_dump_data internal + analyze_parcel try/except wrap)

Backend (parcels.py):
- Import EMPTY_DUMP_RESULT + get_quarter_dump_data
- Call wrapped в try/except — если nspd_quarter_dumps недоступна (DB timeout
  / table missing) → EMPTY_DUMP_RESULT fallback вместо 500 (consistent с
  resilience pattern других optional fetches)
- Response gets 4 new fields:
  - nspd_zoning: dict | None (G1 ПЗЗ — zone_code, zone_name, source)
  - nspd_zouit_overlaps: list[dict] (G3 — overlaps в parcel, per ЗОУИТ group)
  - nspd_engineering_nearby: list[dict] (I3 — engineering structures в 200m)
  - nspd_dump: dict (freshness metadata — available, fetched_at_utc, stale,
    harvest_triggered, total_features)

Tests: 13 new в test_quarter_dump_lookup.py (mock-based, no real DB):
- derive_quarter_cad 5 edge cases (3seg, 4seg, 5seg, invalid, whitespace)
- get_quarter_dump no_row → harvest triggered
- stale (>180d) → harvest triggered, stale=True
- harvest_error row → retry harvest triggered
- parcel_wkt=None → metadata only (1 DB call)
- fresh + zoning extraction
- fresh + zouit_overlaps list
- fresh integration: все 4 keys present

47 pre-existing tests still pass.

Code review (code-reviewer pre-push): MINOR, 0 blocking. Applied 2 of 4:
-  #1: try/except wrap around get_quarter_dump_data в analyze_parcel
  (защита от DB unavailability) + DRY через EMPTY_DUMP_RESULT module const
-  #2: removed redundant nspd_zoning.fetched_at_utc (DRY — freshness в
  nspd_dump.fetched_at_utc)
- ⏭ Deferred (acceptable): #3 ad-hoc harvest_quarter retry cooldown для
  harvest_error rows (только при high traffic + persistent NSPD errors);
  #4 raw_props в response — tech debt, убрать вместе с frontend PR

Performance note: 3 spatial queries per analyze adds ~10-50ms on typical
~100-feature quarter. Mitigation if quarters grow dense: materialized
per-layer sub-table (отдельная DB issue).

Closes Sprint 1.1 part of #94. Frontend rendering этих 4 полей — отдельный
PR (next: #112 / #115 / #114).

* fix(site-finder): address PR #116 auto-review M1-M5

M1 (mutation risk): replace EMPTY_DUMP_RESULT direct refs with
_make_empty_result() factory. dict(...) shallow copy left nested
nspd_dump shared by reference across concurrent requests — single
mutation pollutes module sentinel for all subsequent calls. Now
каждый caller gets independent dict.

M2 (O(N) spatial scan): SELECT extended denormalized counts
(territorial_zones_count, zouit_count, engineering_count). Each
spatial helper accepts layer_counts and early-returns when count=0
— skips heavy jsonb_array_elements + ST_Transform/ST_Intersects
scan entirely. Critical для quarters с 2000+ features.

M4 (documentation): _trigger_harvest docstring describes known
burst/no-dedup limitation + TODO Redis SETNX (отдельный PR).

M5 (test fragility): _make_db_mock_with_spatial docstring describes
positional-call contract — db.execute order (0=dump, 1=zoning, 2=zouit,
3=engineering) и зависимость от count-values.

+4 new tests (17 total, all pass):
- test_make_empty_result_returns_independent_copies (mutation safety)
- test_make_empty_result_overrides
- test_early_exit_all_counts_zero_no_spatial_queries
- test_early_exit_partial_counts

Per auto-review on 3068a9c.

* fix(site-finder): rename _make_empty_result → make_empty_result (public) per PR #116 review

M1 residual fix: parcels.py exception path использовал EMPTY_DUMP_RESULT
singleton ref вместо factory. Сейчас readonly access, но нарушает
documented invariant модуля.

Rename `_make_empty_result` → `make_empty_result` (public API), import в
parcels.py, использовать в try/except fallback. Каждый request получает
независимый dict — никаких shared references.

M4 (Redis SETNX dedup) + M5 (test fragility) — deferred per review,
documented в code/issue. Acceptable trade-offs:
- M4: UPSERT idempotency делает данные safe; burst-duplicate task'и тратят
  WAF traffic впустую но не повреждают данные.
- M5: docstring contractually describes positional-call order.

17/17 tests pass. ruff/format clean.

Per auto-review on aef8308.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-13 09:14:19 +03:00