lekss361/gendesign
1
0
Fork 0
Code Issues 57 Pull requests Projects Releases Packages Wiki Activity Actions
683 commits 226 branches 0 tags 42 MiB
Commit graph

8 commits

Author SHA1 Message Date
lekss361
8191a85102 fix(#231): address re-review blockers — per-instance semaphore + time_limit + test mocks 
PR #260 second-opinion review 🔴 BLOCK → fixes:

1. nspd_bulk_client.py: _SEMAPHORE module-level → per-instance self._sem
   создаётся в __aenter__ под текущий event loop. Sequential asyncio.run()
   в search_by_quarter (11 layers × asyncio.run() per layer) bind'ил
   module-level semaphore к first loop'у → RuntimeError на втором вызове →
   все cells return_exceptions=True проглатывали → grid возвращал 0 features.

2. nspd_sync.py: harvest_quarter добавлен time_limit=900 (hard) рядом с
   soft_time_limit=600. Safety net если task игнорит SoftTimeLimitExceeded.

3. tests/test_nspd_client.py: добавлен _make_fake_grid_walk helper +
   monkeypatch NSPDClient.get_features_in_bbox_grid в 3 тестах
   (search_by_quarter_core_only / _with_zouit / _layers_fetched_with_risks).
   Без этого area layers били бы по живому NSPD API → CI flaky.

Tests: 37 passed, 1 skipped (3 area-dispatch + 34 grid-walk).
 2026-05-17 10:13:34 +03:00
lekss361
fdced94626 fix(#231): switch search_by_quarter area layers to grid-walk dispatch (Sub-PR B) 
- Add _GRID_WALK_LAYERS frozenset (territorial_zones, red_lines,
  engineering_structures, zouit_*, risk_*)
- _fetch_layer dispatch: area layers → get_features_in_bbox_grid(grid_n=7,
  step_m=50) [foundation из PR #247], EGRN layers (parcels, buildings) →
  legacy get_features_in_bbox
- Enrich engineering_structures features с classified_kind через
  classify_engineering_kind (PR #247 classifier)
- Per-layer logger.info heartbeat (method + count) для прогресса
- Bump harvest_quarter soft_time_limit 120 → 600s
  (grid-walk: 11 layers × 49 cells × ~70ms ≈ 40s + retries margin)
- TestFetchLayerDispatch (3 tests): 34 passed, 1 skipped

Pre-push deep-code-reviewer: APPROVE (1 medium nit — opportunity_oopt не в
grid-walk, защищено через include_opportunity=False default; follow-up issue
рекомендован).
 2026-05-17 09:46:42 +03:00
lekss361
992477c8f8 fix(#244): harvest_quarter — CAST CASE WHEN params (psycopg3 AmbiguousParameter) (#246)
Some checks failed
Deploy / build-backend (push) Has been cancelled
Details
Deploy / build-frontend (push) Blocked by required conditions
Details
Deploy / deploy (push) Blocked by required conditions
Details
Deploy / changes (push) Successful in 5s
Details
Deploy / build-worker (push) Has been cancelled
Details
2026-05-17 05:53:08 +00:00
lekss361
db1ff5b1fc feat(nspd): denorm nspd_parcels + nspd_buildings tables (#94 PR3 of 4) 
- data/sql/99_nspd_entities_denorm.sql: 2 tables (cad_num PK) + 6 indexes
  (quarter_cad, GIST geom, partial WHERE purpose ILIKE '%многокв%')
- nspd_denorm.py: denorm_parcel_feature / denorm_building_feature / denorm_dump
  - SAVEPOINT per row via with db.begin_nested()
  - ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON(:geom), 3857), 4326)
  - json.dumps(geom, ensure_ascii=False) for psycopg bind
  - All CAST(:x AS type) — no ❌:type
- nspd_sync.py: inline denorm after _upsert_dump in harvest_quarter (non-fatal)
- nspd_denorm_backfill.py: Celery one-shot task (per-quarter commit)
- admin_etl.py: POST /api/v1/admin/etl/nspd-denorm-backfill (AdminTokenAuth)
- 19 tests: coerce helpers, denorm parcel/building, aggregate counts

Foundation for downstream features (МКД neighbors lookup, parcel attrs).

Part of #94
 2026-05-16 20:20:03 +03:00
lekss361
ed3c128528 feat(nspd): TIER 4 opportunity layers + red lines (#94 PR2 of 4) 
- NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt)
  + QuarterDump.opportunity field
- nspd_sync: harvest_quarter accepts include_opportunity, denorm cols
  has_auction_parcels + opportunity_count in UPSERT
- quarter_dump_lookup:
  - _get_opportunity_parcels (sort by distance, early-exit on count=0)
  - _get_red_lines (query existing dump.red_lines core path, layer='red_lines')
- SQL 89_*: has_auction_parcels + opportunity_count + partial index
- Pydantic: OpportunityParcel + RedLine schemas
- /analyze: nspd_opportunity_parcels + nspd_red_lines fields
- Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration
- Tests: 28 total (11 PR1 + 17 PR2), all pass

Red lines uses existing core harvest path (layer 879243 already in dump.red_lines
from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed.

Part of #94
 2026-05-16 19:06:22 +03:00
lekss361
1595157ebf feat(nspd): enable TIER 3 risk zones harvest + extract in analyze (#94 PR1) 
- harvest_stale_quarters fanout passes include_risks=True to all queued tasks
- _get_risk_zones() with ST_Intersects + ST_Area intersection calc
- _extract_features_by_layer() generic helper for future PR2+ reuse
- RiskZone Pydantic schema; /analyze returns nspd_risk_zones list
- NspdRiskZonesBlock component with severity color-coding (red/yellow/orange)
- 11 unit tests for extract/intersect/null-area/db-exception paths

Part of #94 (TIER 3 risk layers — flooding, landslide, burns, erosion etc)
 2026-05-16 16:11:59 +03:00
lekss361
d104222643
fix(cadastre,nspd_sync): SQL syntax bugs surfaced by ekb_full v2 (#186) 
Three distinct SQL syntax errors caught from worker logs:

1. **upsert_zouit** (bulk_harvest.py:940) — trailing comma after 4326
   in ST_Transform(..., 3857), 4326,) → 'syntax error at or near ")"'.
   Hit on 8+ quarters in 0108 block (66:41:0108055/61/69/77 etc).
   Likely introduced by ruff-format auto-comma on closing paren.

2. **nspd_sync** (nspd_sync.py:120) — `:fetched_at_utc::timestamptz`
   psycopg double-colon trap → 'syntax error at or near ":"'.
   Crashed legacy on-demand harvest_quarter called from analyze.

3. **job_settings** (job_settings.py:224) — `:extra_config::jsonb`
   same double-colon trap in admin settings UPDATE.

All converted to CAST(:x AS type) pattern (psycopg v3 safe).

40 tests still passing.

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-15 19:18:43 +03:00
lekss361
f3b0ce5687
feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4 part A) (#111) 
* feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4/4 part A)

Sprint 1.1 item #3 из #94 part 2. Замыкает harvest cycle: Celery task
оборачивает NSPDClient.search_by_quarter (PR #109) + UPSERT в
nspd_quarter_dumps (PR #110). После — analyze_parcel будет читать dumps
вместо on-demand NSPD HTTP в request-цикле (Sprint 1.1 #4, следующий PR).

Backend:
- New `backend/app/workers/tasks/nspd_sync.py`:
  - `harvest_quarter(quarter_cad, region_code=66, include_zouit=True,
    include_risks=False)` — single-quarter harvest. autoretry_for=
    (NspdLiteWafError,) с retry_backoff=True, max_retries=3,
    soft_time_limit=120s. Generic exception → upsert error row с
    harvest_error=str(exc), counts=0 (НЕ re-raise).
  - `harvest_stale_quarters(region_code, max_age_days=90, batch_size=50)` —
    beat fanout: SELECT cad_quarters_geom EXCEPT fresh dumps → apply_async per cad.
  - UPSERT использует ST_Multi(ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON,3857),4326))
    для quarter_geom (schema MultiPolygon strict per database-expert contract).
- `celery_app.py`:
  - Add `nspd_sync` в `include`
  - Beat entry `nspd-harvest-stale-quarters` Mon 04:00 МСК, batch_size=50,
    max_age_days=90
  - `worker_ready` sanity check `SELECT 1 FROM nspd_quarter_dumps LIMIT 0`
    (non-fatal critical log per Bug_Worker_Ready_EarlyReturn_Fixed pattern)
- `admin_scrape.py`:
  - `HarvestQuarterRequest` Pydantic с pattern `^\d+:\d+:\d+$` (quarter only)
  - `POST /api/v1/admin/scrape/nspd/harvest-quarter` manual trigger,
    requires X-Admin-Token, returns task_id.

Tests (`test_nspd_sync.py`): 14 mock-based (no real NSPD/DB):
- happy path: search_by_quarter mock → verify SQL params/structure
- empty quarter → row with quarter_geom=NULL, total=0, no harvest_error
- WAF error → autoretry kicks in (raise re-propagated)
- Generic error → upsert error row
- Fanout: 50 stale → 50 apply_async calls
- Layer names preservation (all 5 zouit + 11 risks if enabled)
- Geometry SRID handling

Code review (code-reviewer pre-push): APPROVE, 0 blocking, 5 minor deferred:
- inline datetime import in error branch (style)
- layers_fetched array literal via string concat (current layer names safe)
- harvest_stale_quarters error rows always re-tried (intended, undocumented)
- test call_args.kwargs fragile (works for current callsite)
- beat collision with refresh-ekb-districts-medians every ~2 years (negligible)

Part of #94. Next: Sprint 1.1 #4 — analyze_parcel reads dump cache.

* fix(nspd-sync): address PR #111 auto-review M1-M5+M7 findings

M1 (runtime bug): remove WHERE region_code from outer cad_quarters_geom SELECT
in harvest_stale_quarters — column does not exist in prod schema
(cad_number, geom, raw_props, fetched_at, source). Would crash on first beat
tick. Inner nspd_quarter_dumps subquery keeps region_code filter unchanged.

M2 (correctness): _upsert_dump теперь делает db.rollback() в except перед
db.close() — prevents PendingRollbackError на следующем session use, если
PostGIS отбракует geometry или JSON cast failure.

M3 (low): worker_ready breadcrumb INSERT тоже теперь имеет rollback —
consistency с остальными session usages в celery_app.py.

M4 (style): layers_fetched — заменил manual string concat
'{' + ','.join(...) + '}' на native Python list. psycopg v3 сам сериализует
list → text[]. Защита от future layer names с запятыми/фигурными скобками.

M5 (style): import datetime as _dt поднят из else branch в top-level imports.

M7 (test cleanup): удалил dead __enter__/__exit__ mock setup в
test_harvest_stale_quarters_fanout / _empty. harvest_stale_quarters не
использует session как context manager — moc'и no-op.

14/14 tests pass. ruff/format clean.

Per auto-review on 9b2a289.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
 2026-05-12 18:45:57 +03:00