78 commits

Author	SHA1	Message	Date
lekss361	396db20c48	feat(sf): enumerate изъятия/резервирования с pravo.gov66 → land_reservation (#1145) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 13:56:29 +00:00
Light1YT	1f66dfd400	feat(macro): ЕМИСС среднедушевые доходы (id=57039) -> macro_indicator (#946 part2) ... Расширяет Росстат-скрейпер ЕМИСС/fedstat-рядом «Среднедушевые денежные доходы населения» (fedstat id=57039, OKATO 65=Свердл, квартальный, руб). Добавляет pure SDMX-ML v1.0 GenericData-парсер (stdlib ElementTree, dev-тестируем на фикстурах) + EMISS-ветку Celery-таски rosstat_macro_sync (open-data + ЕМИСС, per-source guard, source='emiss', frequency параметризован, CAST not ::, SAVEPOINT per-row). income_per_capita проброшен в site_finder/macro (region='sverdl'). ЕМИСС за WAF с dev -> fetch исполняется на проде (verified: httpx POST из gendesign-worker-1 -> 200 + SDMX, парсер извлёк 9 Свердл-строк). ИПЦ (id=31074) / индекс цен СМР НЕ landed: многомерные, дефолт-экспорт data.do без Свердл, нужна dataGrid-выборка (selectedFilterIds + filter-tree AJAX + reCAPTCHA) -- задокументировано. Refs #946.	2026-06-07 18:44:38 +05:00
Light1YT	f7c4d7a8c5	feat(macro): Росстат open-data macro scraper -> macro_indicator (#946) ... EPIC2 macro-ingest: региональный (Свердловская обл. ОКТМО 65 / ЕКБ) скрейпер Росстата в унифицированную macro_indicator (м.123). Зеркалит форму cbr_macro: pure-парсеры + тонкий httpx, Celery-таска с SAVEPOINT-per-row upsert (CAST not ::, ON CONFLICT по PK indicator_type/region/obs_date). ЕМИСС/fedstat.ru (SDMX) за WAF (hard-403 на всех путях из dev-IP) -> ИПЦ/доходы/ СМР-цены пока недоступны (документировано в коде). rosstat.gov.ru/opendata (стандарт 4.0) WAF-free -> приземляет демографию population_total (sverdl 4.32M / ekb 1.41M, §7.11). - services/scrapers/rosstat_emiss.py — fetch+parse open-data (registry->meta->data CSV) - workers/tasks/rosstat_macro_sync.py — Celery upsert (source=rosstat) - beat: rosstat-macro-sync-monthly; include в celery_app - site_finder/macro.py: population_total -> region-aware default sverdl (additive) - tests: 18 offline-тестов парсера + контракта таски Без миграции (вписано в существующую схему), без новых зависимостей. Refs #946.	2026-06-07 13:12:01 +00:00
lekss361	50fcba1ca0	feat(sf): ingest ОКН-объектов ЕКБ из АИС ЕГРКН (точки+категория) → okn_objects (#1141) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 12:51:18 +00:00
bot-backend	86b9a5a197	perf(sf): КРТ-геометрия в БД, _krt_at из БД вместо живого WFS (#1139) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-06-07 12:14:35 +00:00
lekss361	189ecc81f8	feat(sf): ingest функц.зон генплана ЕКБ-2045 из ГИСОГД-СО WFS → ekb_genplan_functional_zone (#1137) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 12:03:39 +00:00
lekss361	1e53416277	feat(sf): ППТ/ПМТ ТЭП-парсер (Табл.11/12/13) → ekb_ppt_tep MVP (#1133) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 11:54:40 +00:00
lekss361	57c6d119f5	feat(sf): scraper реестра КРТ-площадок ЕКБ → ekb_krt_sites (#1126) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 11:24:49 +00:00
lekss361	ef15190690	feat(sf): geocode geom для ekburg_construction_permits из МСК-66 (#1124) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 11:16:45 +00:00
lekss361	59f2628e0b	feat(sf): ПАГЕ-парсер изъятия/резервирования → land_reservation (#1118) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 10:15:03 +00:00
lekss361	dcafb32f31	feat(sf): gknspecial_zone ИРД-harvest → ird_overlays (#1114) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 09:54:53 +00:00
lekss361	fe7727e338	fix(sf): per-quarter commit в ird/opportunity harvest — не терять прогресс при краше (#1107) ... ird_harvest + opportunity_harvest: commit после КАЖДОГО квартала (внутри per-quarter try) + db.rollback() в except перед continue. Прод-баг: single end-commit терял весь прогресс многочасового grid-walk при краше/таймауте/WorkerLost. Теперь durable поквартально; SAVEPOINT per-row не тронут. +2 теста (commit-count durability, flaky-quarter rollback+continue). Refs #1067. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 21:47:35 +00:00
lekss361	dee7d64ac8	feat(sf): прогрев zone_regulation_cache по терзонам ЕКБ (#1106) ... zone_regulation_refresh.py — Celery refresh_zone_regulations: WFS-перечисление терзон ЕКБ → дедуп по urban_index → shapely.representative_point → zone_regulation_at (urbanCard) → upsert в zone_regulation_cache (м.133). Проактивный прогрев кэша → analyze отдаёт ird.zone_regulation из кэша без live-urbanCard в hot-пути (снимает latency #1102). beat monthly (1-е 06:30). Без миграции. Зависит от #1103. 4 теста. Refs #1059. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 21:04:52 +00:00
lekss361	9e4348629b	feat(sf): ППТ/ПМТ WFS ingest → planning_projects (future-supply, #1104) ... Миграция 134_planning_projects.sql + planning_harvest.py: ингест утверждённых ППТ/ПМТ ЕКБ из геопортал-WFS (один BBOX-запрос на слой, geom уже 4326 → SetSRID без Transform) → planning_projects (project_type/source_key UNIQUE, doc_status/год). SAVEPOINT per-row, beat monthly (1-е 06:00). Зависит от TLS-фикса #1103. Документ-центричная таблица отдельно от ird_overlays. 3 теста. ТЭП/PDF — follow-up. Refs #1085. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 20:25:23 +00:00
lekss361	b1530467f7	feat(sf): opportunity-ЗУ harvest → ird_overlays (future-supply, #1100) ... opportunity_harvest.py — grid-walk НСПД opportunity-слоёв (ЗУ аукцион/схема/свободные/межевание/ККР) по кварталам ЕКБ → ird_overlays с layer_kind='opportunity_*'. Переиспользует движок ird_harvest (#1090): _upsert_feature/_ekb_quarters. +krt_territories=37430 в LAYERS. Beat monthly (1-е 05:30). Self-contained, прод-прогон gated B6. Closes #1086. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 19:32:45 +00:00
lekss361	a15265c594	feat(sf): ИРД-harvest Celery task → ird_overlays (#1090) ... ird_harvest.py — периодический harvest готовых НСПД ИРД-слоёв (ЗОУИТ/ПЗЗ/ОКН/ООПТ/лесничества/береговые) по кварталам ЕКБ → upsert в ird_overlays (м.132) по (source_layer_id, geom_data_id). SAVEPOINT per-feature, content_* не перетирается. Beat ird-harvest-weekly (Mon 05:00 МСК) + celery include. Keystone overlay-движка #1067 B5; прод-прогон деферится в B6 smoke. Refs #1067. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 18:52:04 +00:00
Light1YT	c94810f3a7	feat(forecast): forecast_site_finder_report Celery task + schema-filtered run read (3b-i) ... build_site_finder_report (§22) takes ~30-180s → runs in a background Celery task, not inline on the sync /analyze endpoint. - repository: latest_run_for gains keyword-only schema_version (default None keeps v_analysis_runs_latest behavior, backward-compat); when given, reads base analysis_runs filtered by schema_version ORDER BY created_at DESC LIMIT 1 — fetches the latest analyze-1.0 site-analysis run even when newer 1.0 (§22) rows exist on top (index-served via 127's (cad_num, created_at DESC)). - new workers/tasks/forecast.py::forecast_site_finder_report: reads latest analyze-1.0, runs the §22 orchestrator, persists SiteFinderReport.as_dict() as a 1.0 run via persist_analysis_run. Graceful: no base run / compute error → logger + return None (worker not crashed). time_limit=900/soft=840 (no global limit). Registered in include. Prod-confirmed: analyze-1.0 result carries the full analyze dict (competitors+district) → orchestrator input valid. Endpoint trigger (3b-ii) + §9.x untouched. 943 tests pass; code-review APPROVE (contracts verified vs real as_dict(); status done→complete normalized, no IntegrityError). Refs #994 #961.	2026-06-05 09:02:15 +05:00
Light1YT	5f60668642	fix(nspd-geo): coerce geom to MultiPolygon + wide cad_buildings schema in on-demand ingest ... REAL code-level cause (prod-confirmed, 2 parcels: "Geometry type (Polygon) does not match column type (MultiPolygon)") of single-contour parcels stuck "fetching" — complements migrations 129/130 (which fixed the cad_parcels NOT NULL chain). F1: _save_parcel/_save_quarter now ST_Multi() the geom so single-contour Polygons fit the MultiPolygon columns (migr 93/58), mirroring the bulk_harvest paths. F2: _save_building rewritten to the wide cad_buildings schema (migr 92): quarter_cad_number (was quarter_cad_num → UndefinedColumn), _safe_int(floors) (range strings → NULL), source='nspd' NOT NULL. F3: upsert_features wraps each feature in begin_nested() SAVEPOINT so one bad feature no longer aborts the whole quarter snapshot (backend.md rule; mirrors grid-walk). New tests/workers/test_nspd_geo.py + savepoint tests. 89 passed.	2026-06-04 13:11:47 +05:00
Light1YT	ed3b302d57	fix(supply-layers): thread dev_group_name into L3 upsert key (#970 CRITICAL) ... REOPENED. L3 future-supply rows are computed per (district_name, dev_group_name) but dev_group_name was never a key column — only embedded in method text. With complex_id/obj_class NULL for L3, every dev_group of a district collapsed to one upsert key → ~95.6% loss. Ground-truth (Академический, prod): should be 13,808 units / 15 dev_groups / 54 objects; only 1 row / 607 units survived. Migration 128: ADD COLUMN supply_layers.dev_group_name TEXT + rebuild uq_supply_layers_logical to (layer, district_name, complex_id, obj_class, dev_group_name, source, snapshot_date) NULLS NOT DISTINCT (L1/L2 dev_group_name NULL stays transparent → their dedup unchanged; L3 distinct groups no longer collapse). Dry-run-verified vs prod catalog (applies clean, ROLLBACK clean). Worker: SupplyLayerRow gains dev_group_name (L1/L2=None, L3=group); _UPSERT_SQL adds it to INSERT/VALUES (CAST(:dev_group_name AS text)) + ON CONFLICT (key col, not in DO UPDATE SET). Service+worker regression tests assert same-district/ different-dev_group → distinct keys (no collapse). 234 supply tests pass. Deploy applies migration before container restart; collapsed data self-heals on next supply_layers_refresh. Verification = prod re-measure post-deploy. Refs #970	2026-06-04 10:06:30 +05:00
Light1YT	ae85f19993	feat(workers): supply-layers refresh task + weekly beat (#950 Step 5) (#1005)	2026-06-02 20:47:05 +00:00
Light1YT	c78c6ec2d0	feat(site-finder): live CBR key-rate scraper → macro_indicator (#945 PR B) (#964)	2026-06-02 19:05:57 +00:00
bot-backend	87a5de0cae	feat(db): quarter_price_index FDW foreign table + monthly refresh (Refs #762) (#797) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 17:16:28 +00:00
lekss361	44cb5b509d	fix(beat): distinguish DB-unreachable from intentional-empty schedule (#520)	2026-05-24 13:14:11 +00:00
lekss361	ac57b8e905	fix(scrapers): disable catalog-object SSR beat-cron — DOM.РФ WAF cooldown (#516)	2026-05-24 12:21:10 +00:00
lekss361	f9adb5cc2f	feat(22d): catalog scraper — skip-today filter + force flag (#348)	2026-05-17 23:15:22 +00:00
lekss361	96b03872ac	fix(22d): register scrape_kn_catalog_objects in celery include list (#339)	2026-05-17 21:54:13 +00:00
lekss361	0567ad2130	feat(22d): domrf_catalog_object scraper — fill ~25 NULL kn_objects cols from SSR __NEXT_DATA__ (#335)	2026-05-17 21:26:22 +00:00
lekss361	389e141ec9	fix(scrape-objective): stream-parse lots_pf via ijson to avoid OOM (#315)	2026-05-17 16:37:30 +00:00
lekss361	fce48a8924	fix(scrape-objective): drop redundant self arg in sync_all_groups loop (#313)	2026-05-17 16:13:41 +00:00
lekss361	d4da1b26c6	fix(#231): re-enable nspd-harvest-stale-quarters beat schedule (Sub-PR D) ... После Sub-PR B (#260, merged 8ebcd15) переключения search_by_quarter на grid-walk dispatch — beat schedule безопасно re-enable'ить: - harvest_quarter возвращает полноценные dumps (territorial_zones + ЗОУИТ + risk zones + engineering structures) вместо 0-3 features - per-instance semaphore (PR #260) защищает от cross-loop binding bug - soft/hard time_limit 600/900s — задача успевает завершиться Schedule: Mon 01:00 UTC (04:00 МСК), batch_size=50 ограничивает WAF burst, max_age_days=90 для регулярного refresh + новые quarters (fetched_at IS NULL). Полный ЕКБ back-fill (11k+ quarters в cad_quarters_geom) займёт несколько недель при batch_size=50/week; в production будет ускорено manual trigger admin endpoint'ом если нужно. Closes #231.	2026-05-17 10:37:07 +03:00
lekss361	8191a85102	fix(#231): address re-review blockers — per-instance semaphore + time_limit + test mocks ... PR #260 second-opinion review 🔴 BLOCK → fixes: 1. nspd_bulk_client.py: _SEMAPHORE module-level → per-instance self._sem создаётся в __aenter__ под текущий event loop. Sequential asyncio.run() в search_by_quarter (11 layers × asyncio.run() per layer) bind'ил module-level semaphore к first loop'у → RuntimeError на втором вызове → все cells return_exceptions=True проглатывали → grid возвращал 0 features. 2. nspd_sync.py: harvest_quarter добавлен time_limit=900 (hard) рядом с soft_time_limit=600. Safety net если task игнорит SoftTimeLimitExceeded. 3. tests/test_nspd_client.py: добавлен _make_fake_grid_walk helper + monkeypatch NSPDClient.get_features_in_bbox_grid в 3 тестах (search_by_quarter_core_only / _with_zouit / _layers_fetched_with_risks). Без этого area layers били бы по живому NSPD API → CI flaky. Tests: 37 passed, 1 skipped (3 area-dispatch + 34 grid-walk).	2026-05-17 10:13:34 +03:00
lekss361	fdced94626	fix(#231): switch search_by_quarter area layers to grid-walk dispatch (Sub-PR B) ... - Add _GRID_WALK_LAYERS frozenset (territorial_zones, red_lines, engineering_structures, zouit_*, risk_*) - _fetch_layer dispatch: area layers → get_features_in_bbox_grid(grid_n=7, step_m=50) [foundation из PR #247], EGRN layers (parcels, buildings) → legacy get_features_in_bbox - Enrich engineering_structures features с classified_kind через classify_engineering_kind (PR #247 classifier) - Per-layer logger.info heartbeat (method + count) для прогресса - Bump harvest_quarter soft_time_limit 120 → 600s (grid-walk: 11 layers × 49 cells × ~70ms ≈ 40s + retries margin) - TestFetchLayerDispatch (3 tests): 34 passed, 1 skipped Pre-push deep-code-reviewer: APPROVE (1 medium nit — opportunity_oopt не в grid-walk, защищено через include_opportunity=False default; follow-up issue рекомендован).	2026-05-17 09:46:42 +03:00
lekss361	992477c8f8	fix(#244): harvest_quarter — CAST CASE WHEN params (psycopg3 AmbiguousParameter) (#246)	2026-05-17 05:53:08 +00:00
lekss361	232dbc9285	fix(#233): add pzz_sync to beat schedule (monthly) ... PKK6 ArcGIS sync для pzz_zones_ekb (0 rows live) — задача и admin endpoint уже существуют, но beat schedule отсутствовал. Schedule: 1st of month, 03:00 МСК. Closes #233	2026-05-17 08:29:40 +03:00
lekss361	db1ff5b1fc	feat(nspd): denorm nspd_parcels + nspd_buildings tables (#94 PR3 of 4) ... - data/sql/99_nspd_entities_denorm.sql: 2 tables (cad_num PK) + 6 indexes (quarter_cad, GIST geom, partial WHERE purpose ILIKE '%многокв%') - nspd_denorm.py: denorm_parcel_feature / denorm_building_feature / denorm_dump - SAVEPOINT per row via with db.begin_nested() - ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON(:geom), 3857), 4326) - json.dumps(geom, ensure_ascii=False) for psycopg bind - All CAST(:x AS type) — no ❌:type - nspd_sync.py: inline denorm after _upsert_dump in harvest_quarter (non-fatal) - nspd_denorm_backfill.py: Celery one-shot task (per-quarter commit) - admin_etl.py: POST /api/v1/admin/etl/nspd-denorm-backfill (AdminTokenAuth) - 19 tests: coerce helpers, denorm parcel/building, aggregate counts Foundation for downstream features (МКД neighbors lookup, parcel attrs). Part of #94	2026-05-16 20:20:03 +03:00
lekss361	5a7d405c77	Merge pull request 'feat(nspd): TIER 4 opportunity + red lines (#94 PR2 of 4)' (#220) from feat/94-pr2-opportunity-redlines into main	2026-05-16 17:09:52 +00:00
lekss361	ed3c128528	feat(nspd): TIER 4 opportunity layers + red lines (#94 PR2 of 4) ... - NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt) + QuarterDump.opportunity field - nspd_sync: harvest_quarter accepts include_opportunity, denorm cols has_auction_parcels + opportunity_count in UPSERT - quarter_dump_lookup: - _get_opportunity_parcels (sort by distance, early-exit on count=0) - _get_red_lines (query existing dump.red_lines core path, layer='red_lines') - SQL 89_*: has_auction_parcels + opportunity_count + partial index - Pydantic: OpportunityParcel + RedLine schemas - /analyze: nspd_opportunity_parcels + nspd_red_lines fields - Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration - Tests: 28 total (11 PR1 + 17 PR2), all pass Red lines uses existing core harvest path (layer 879243 already in dump.red_lines from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed. Part of #94	2026-05-16 19:06:22 +03:00
lekss361	b61f025673	feat(backend): sentry-sdk init для FastAPI + Celery (#204 backend) (#207)	2026-05-16 15:09:16 +00:00
lekss361	1595157ebf	feat(nspd): enable TIER 3 risk zones harvest + extract in analyze (#94 PR1) ... - harvest_stale_quarters fanout passes include_risks=True to all queued tasks - _get_risk_zones() with ST_Intersects + ST_Area intersection calc - _extract_features_by_layer() generic helper for future PR2+ reuse - RiskZone Pydantic schema; /analyze returns nspd_risk_zones list - NspdRiskZonesBlock component with severity color-coding (red/yellow/orange) - 11 unit tests for extract/intersect/null-area/db-exception paths Part of #94 (TIER 3 risk layers — flooding, landslide, burns, erosion etc)	2026-05-16 16:11:59 +03:00
lekss361	46203ab365	feat(permits): Celery beat monthly + analyze recent_permits (#105 Phase 4+5) (#213)	2026-05-16 12:23:52 +00:00
lekss361	adc45f5724	feat(permits): ЕКБ РНС/РВЭ xlsx parser + schema + ETL (#105 PR A) (#211)	2026-05-16 11:41:21 +00:00
lekss361	4d492ebc50	perf(cadastre): freshness skip + grid-walk heartbeat + log noise reduction (#187) ... * perf(cadastre): freshness skip + grid-walk heartbeat + reduce WMS log noise Three operational improvements after ekb_full v3 completed at 99.1%: Fix H — per-quarter freshness skip (scrape_cadastre.py): - enqueue_cadastre_harvest queries cad_quarter_stats.fetched_at per quarter - Skip quarters where fetched_at >= NOW() - skip_fresh_hours (default 1h) - extra_config.skip_fresh_hours=0 disables skip (full re-fetch) - Saves ~30k NSPD calls on repeated runs of same scope Fix I — heartbeat in grid-walk loop (bulk_harvest.py:_grid_walk_category): - 225 cells x 70-300ms = 15-75s heartbeat silence - Pass update_progress callback to _grid_walk_category - Call every heartbeat_every (50) cells with phase progress info - Prevents cleanup_zombies false-positive cancellation Fix J — NSPD WMS HTTP 500 noise to debug (bulk_harvest.py:_grid_walk_category): - NSPD WMS endpoint frequently returns HTTP 500 (server-side issue) - Was: logger.warning each, 200+ warnings per quarter - Now: logger.debug (visible via --log-level=DEBUG) 40 tests passing. * fix(cadastre): parse NSPD DD.MM.YYYY dates (Fix K — job 18 root cause) Job 18 (manual_list recovery 11 quarters) crashed every quarter with: psycopg.errors.DatetimeFieldOverflow: date/time field value out of range: '13.03.2023' NSPD returns registration_date in mixed formats: - DD.MM.YYYY for Сооружения / ОНС / ЕНК / ЗОУИТ - YYYY-MM-DD ISO for ЗУ / Здания - YYYY-MM-DD HH:MM:SS with time PG CAST(:date AS date) accepts ISO only — DD.MM.YYYY crashes. Add _parse_nspd_date helper: handles both formats + None/empty. Apply in upsert_construction, upsert_onc, upsert_enk, upsert_zouit, upsert_parcel for registration_date / build_record_registration_date / legal_act_date. 3 new unit tests for the helper (43 tests total). * fix(cadastre): address bot blockers — pivot Fix H to raw_targets + tests Bot review on PR #187 caught: 1. cadastre_jobs has no extra_config column → Fix H crashed on KeyError 2. Missing tests for Fix H + Fix I 3. _parse_nspd_date didn't validate date ranges Changes: - Pivot Fix H: read skip_fresh_hours from raw_targets JSONB (existing column), no schema migration needed - _parse_nspd_date: validate via date.fromisoformat() — rejects 2023-13-45 - Add 3 tests for Fix H in backend/tests/workers/test_scrape_cadastre.py - Add 2 tests for Fix I heartbeat callbacks in test_cadastre_bulk.py - Add 1 test for parse_nspd_date range validation (was: only format) 49 tests passing. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 23:02:37 +03:00
lekss361	d104222643	fix(cadastre,nspd_sync): SQL syntax bugs surfaced by ekb_full v2 (#186) ... Three distinct SQL syntax errors caught from worker logs: 1. **upsert_zouit** (bulk_harvest.py:940) — trailing comma after 4326 in ST_Transform(..., 3857), 4326,) → 'syntax error at or near ")"'. Hit on 8+ quarters in 0108 block (66:41:0108055/61/69/77 etc). Likely introduced by ruff-format auto-comma on closing paren. 2. **nspd_sync** (nspd_sync.py:120) — `:fetched_at_utc::timestamptz` psycopg double-colon trap → 'syntax error at or near ":"'. Crashed legacy on-demand harvest_quarter called from analyze. 3. **job_settings** (job_settings.py:224) — `:extra_config::jsonb` same double-colon trap in admin settings UPDATE. All converted to CAST(:x AS type) pattern (psycopg v3 safe). 40 tests still passing. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 19:18:43 +03:00
lekss361	057f8891dc	fix(cadastre): count snapshot+per-cat requests in cadastre_jobs.requests_count (#182) ... scrape_cadastre.py previously incremented requests_count only by grid_walk_requests, so snapshot-only quarters (and the entire ekb_full job after PR #179 skips grid-walk on broken geom) reported 0 NSPD requests — misleading for observability. Add HarvestResult.snapshot_requests counter (Phase 1 search + Phase 1.5 per-cat probes) and a total_requests property. Update Celery task to write total_requests into cadastre_jobs.requests_count. Tests added for the new property + Phase 1 snapshot_requests=1 assertion. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 17:17:01 +03:00
lekss361	5da66bbc93	feat(cadastre): bulk_harvest worker + grid-walker + admin API (#168 PR3/5) (#171) ... * feat(cadastre): bulk_harvest_quarter Celery task + grid-walker + saga state (#168 PR3/5) Add bulk cadastre harvest pipeline: - services/cadastre/bulk_harvest.py: async harvest_quarter() orchestrator (4 phases) + 7 upsert helpers (parcels/buildings/constructions/oncs/enks/zouit/quarter_stats) using CAST(:x AS jsonb) pattern, begin_nested() SAVEPOINT per grid-walk upsert - services/cadastre/grid_geometry.py: quarter_bbox_3857 (PostGIS ST_Extent) + generate_grid_click_points (15x15 = 225 sub-bbox grid) - workers/tasks/scrape_cadastre.py: bulk_harvest_quarter_task (acks_late=True, dont_autoretry_for NspdBulkWafError), enqueue_cadastre_harvest, cleanup_zombies - api/v1/admin_cadastre.py: 5 endpoints behind AdminTokenAuth — create/list/get/cancel/resume - Tests: 13 service unit + 8 API tests * fixup(cadastre): drop importorskip (PR2 merged) — imports now top-level (#168 PR3) * fixup(cadastre): tile_size unified param + register slow marker (#168 PR3) Blocker #1: rename tile_width/tile_height → tile_size in generate_grid_click_points. Callers in bulk_harvest.py and test_cadastre_bulk.py used tile_size; def had tile_width+tile_height → TypeError at runtime. Blocker #2 (10 failures in test_admin_cadastre.py) was side-effect of #1: TypeError at import chain (app.main → admin_cadastre → bulk_harvest) broke FastAPI app load → dependency_overrides AttributeError. Now resolved. Also register `slow` pytest marker in pyproject.toml to suppress PytestUnknownMarkWarning. * fixup(cadastre): use importlib.util.find_spec instead of import (#168 PR3) Root cause Blocker #2: `import app.workers.tasks.scrape_cadastre` at module top-level REBOUND `app` from FastAPI instance (line 14) to the Python package. All subsequent `app.dependency_overrides[get_db] = ...` failed with AttributeError because `app` was now the namespace module. Fix: probe module availability with importlib.util.find_spec — does not import or rebind names. FastAPI `app` instance stays intact. All 28 cadastre tests pass locally. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 13:31:32 +03:00
lekss361	6a256bfc1f	refactor(workers): split celery_app.py god-object into 3 modules (#135) ... Per audit batch #127 P3 god objects (issue #134). Split layout: - celery_app.py: 270 → 35 lines (config only) - beat_schedule.py: 207 lines NEW (DB-driven + fallback + hardcoded) - lifecycle.py: 168 lines NEW (worker_process_init + worker_ready handlers) No business logic changes — move-only structural refactor. Public API (celery_app.conf.beat_schedule) unchanged. Verify: - beat keys: kn-region-66, objective_sync, refresh-ekb-districts-medians, nspd-geo-zombie-cleanup, poi-sync-weekly, noise-sync-weekly - ruff + ruff-format clean - AST parse OK Vault: Module_Beat_Schedule + Module_Worker_Lifecycle NEW; Module_Celery_App updated. Closes #134 Refs: #127 Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 00:09:38 +03:00
lekss361	8dfab7613d	fix(nspd-geo): replace f-string bulk INSERT with parametrized batch execute (#123) ... Per code review audit (May 14): real SQL injection vector в `enqueue_geo_job`. Manual `chr(39)*2` quote escape НЕ injection-safe — не защищает от backslash escape, Unicode quote tricks. Если cad_num когда-либо придёт из user input — exploitable. ## Change `backend/app/workers/tasks/nspd_geo.py:298-310` — заменил f-string VALUES concatenation на SQLAlchemy `text(...)` + list of dicts (psycopg v3 handles batch executemany под капотом). Pattern consistent с остальными INSERT'ами в этом же файле (_log, _start_job, _save_quarter, _save_parcel, _save_building). ## Preserved - ON CONFLICT (job_id, cad_num, thematic_id) DO NOTHING — verbatim - Return value (int(job_id)) - Surrounding tx/commit/close logic - DB schema, function signature, caller interface — unchanged ## Tests - No test_nspd_geo.py существует — skipped - ruff check passed ## Vault `fixes/Bug_Nspd_Geo_Sql_Injection_May14.md` — created (status: resolved). No backward compat / migration concerns — pure in-place query rewrite. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-14 23:29:12 +03:00
lekss361	f5794173de	fix(celery): disable nspd-harvest-stale-quarters beat (Bug post-mortem #1) (#121) ... Per fixes/Bug_NSPD_WMS_NotBulk_2026_May14 action item #1: отключить beat schedule `nspd-harvest-stale-quarters` до Sprint 2 fix (grid sampling rewrite). ## Why harvest_quarter в текущей реализации пишет почти пустые dumps в nspd_quarter_dumps из-за single-pixel WMS GetFeatureInfo bug (детали в vault post-mortem). Beat запуск Mon 04:00 МСК (next: 2026-05-18 04:00 МСК): - Потратит rate-limit budget на ~12k EKB кварталов - Заполнит nspd_quarter_dumps мусорными rows - analyze_parcel начнёт отдавать пустые nspd_zoning/nspd_zouit_overlaps/ nspd_engineering_nearby для всех пользователей (через quarter_dump_lookup) ## What changed `backend/app/workers/celery_app.py` — закомментирован `schedule[ "nspd-harvest-stale-quarters"]` блок (~10 строк) с inline-комментом explanation + ссылкой на vault post-mortem. ## What НЕ changed - `backend/app/workers/tasks/nspd_sync.py` — task code (harvest_quarter, harvest_stale_quarters) остаётся, можно re-enable после Sprint 2 fix - Admin endpoint для manual trigger остаётся работающим - Все остальные beat tasks (poi-sync, noise-sync, zombie-cleanup, refresh-ekb-districts-medians, kn-region-*) — не тронуты ## Vault - fixes/Bug_NSPD_WMS_NotBulk_2026_May14.md — status `identified` → `mitigated` (уже обновлено через vault audit pass) ## Re-enable После Sprint 2 grid-sampling rewrite (см. Bug post-mortem → Sprint 2 fix-strategy) — uncomment блок + verify через staging. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-14 23:24:53 +03:00
lekss361	f3b0ce5687	feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4 part A) (#111) ... * feat(nspd): harvest_quarter Celery task + beat + admin endpoint (#94 pt.4/4 part A) Sprint 1.1 item #3 из #94 part 2. Замыкает harvest cycle: Celery task оборачивает NSPDClient.search_by_quarter (PR #109) + UPSERT в nspd_quarter_dumps (PR #110). После — analyze_parcel будет читать dumps вместо on-demand NSPD HTTP в request-цикле (Sprint 1.1 #4, следующий PR). Backend: - New `backend/app/workers/tasks/nspd_sync.py`: - `harvest_quarter(quarter_cad, region_code=66, include_zouit=True, include_risks=False)` — single-quarter harvest. autoretry_for= (NspdLiteWafError,) с retry_backoff=True, max_retries=3, soft_time_limit=120s. Generic exception → upsert error row с harvest_error=str(exc), counts=0 (НЕ re-raise). - `harvest_stale_quarters(region_code, max_age_days=90, batch_size=50)` — beat fanout: SELECT cad_quarters_geom EXCEPT fresh dumps → apply_async per cad. - UPSERT использует ST_Multi(ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON,3857),4326)) для quarter_geom (schema MultiPolygon strict per database-expert contract). - `celery_app.py`: - Add `nspd_sync` в `include` - Beat entry `nspd-harvest-stale-quarters` Mon 04:00 МСК, batch_size=50, max_age_days=90 - `worker_ready` sanity check `SELECT 1 FROM nspd_quarter_dumps LIMIT 0` (non-fatal critical log per Bug_Worker_Ready_EarlyReturn_Fixed pattern) - `admin_scrape.py`: - `HarvestQuarterRequest` Pydantic с pattern `^\d+:\d+:\d+$` (quarter only) - `POST /api/v1/admin/scrape/nspd/harvest-quarter` manual trigger, requires X-Admin-Token, returns task_id. Tests (`test_nspd_sync.py`): 14 mock-based (no real NSPD/DB): - happy path: search_by_quarter mock → verify SQL params/structure - empty quarter → row with quarter_geom=NULL, total=0, no harvest_error - WAF error → autoretry kicks in (raise re-propagated) - Generic error → upsert error row - Fanout: 50 stale → 50 apply_async calls - Layer names preservation (all 5 zouit + 11 risks if enabled) - Geometry SRID handling Code review (code-reviewer pre-push): APPROVE, 0 blocking, 5 minor deferred: - inline datetime import in error branch (style) - layers_fetched array literal via string concat (current layer names safe) - harvest_stale_quarters error rows always re-tried (intended, undocumented) - test call_args.kwargs fragile (works for current callsite) - beat collision with refresh-ekb-districts-medians every ~2 years (negligible) Part of #94. Next: Sprint 1.1 #4 — analyze_parcel reads dump cache. * fix(nspd-sync): address PR #111 auto-review M1-M5+M7 findings M1 (runtime bug): remove WHERE region_code from outer cad_quarters_geom SELECT in harvest_stale_quarters — column does not exist in prod schema (cad_number, geom, raw_props, fetched_at, source). Would crash on first beat tick. Inner nspd_quarter_dumps subquery keeps region_code filter unchanged. M2 (correctness): _upsert_dump теперь делает db.rollback() в except перед db.close() — prevents PendingRollbackError на следующем session use, если PostGIS отбракует geometry или JSON cast failure. M3 (low): worker_ready breadcrumb INSERT тоже теперь имеет rollback — consistency с остальными session usages в celery_app.py. M4 (style): layers_fetched — заменил manual string concat '{' + ','.join(...) + '}' на native Python list. psycopg v3 сам сериализует list → text[]. Защита от future layer names с запятыми/фигурными скобками. M5 (style): import datetime as _dt поднят из else branch в top-level imports. M7 (test cleanup): удалил dead __enter__/__exit__ mock setup в test_harvest_stale_quarters_fanout / _empty. harvest_stale_quarters не использует session как context manager — moc'и no-op. 14/14 tests pass. ruff/format clean. Per auto-review on 9b2a289. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-12 18:45:57 +03:00
lekss361	8be539ddc6	feat(site-finder): ПЗЗ territorial zones from Rosreestr PKK6 + zoning in analyze	2026-05-11 21:51:51 +03:00