50 commits

Author	SHA1	Message	Date
bot-backend	12162a0abb	feat(exporters): Site Finder report PDF exporter (#989, 955-A3) (#1023)	2026-06-03 09:09:30 +00:00
bot-backend	299f2365fb	feat(exporters): Excel exporter for SiteFinderReport (#991, 955-A5) (#1022)	2026-06-03 08:57:00 +00:00
bot-backend	fc312fe606	feat(forecasting): §13 report assembler (#988, 955-A2) (#1021)	2026-06-03 08:51:23 +00:00
bot-backend	47fe5a55eb	feat(forecasting): §15 confidence engine v2 (#990, 955-A4) (#1020)	2026-06-03 08:41:07 +00:00
bot-backend	75c9b34987	feat(forecasting): §13 SiteFinderReport object (#987, 955-A1) (#1019)	2026-06-03 08:33:01 +00:00
bot-backend	008f9d9ed1	feat(forecasting): §25 six special indices (#986, 954-C) (#1018)	2026-06-03 08:24:08 +00:00
bot-backend	98dea0315a	feat(forecasting): §14.2 product scoring card (#985, 954-B) (#1017)	2026-06-03 08:11:54 +00:00
bot-backend	8057468c13	feat(forecasting): §11 macro-scenarios (#984, 954-A) (#1016)	2026-06-03 07:59:22 +00:00
bot-backend	691ccef4b7	feat(forecasting): class/commercial/USP §10.2/10.4/10.5 + §16 (#983, 953-B) (#1015)	2026-06-03 07:52:24 +00:00
bot-backend	72e9b24f2c	feat(recommend): horizon-aware recommend_mix opt-in overlay (#982, 953-A) (#1014)	2026-06-03 07:40:33 +00:00
bot-backend	489e380f1e	feat(forecasting): what-to-build ranker + MAI proxy (#981, 952-B) (#1013)	2026-06-03 07:07:17 +00:00
bot-backend	70ffa399fc	feat(forecasting): demand-supply forecast engine (#980, 952-A) (#1012)	2026-06-03 06:56:33 +00:00
bot-backend	9758e21cbd	feat(forecasting): §9.4 demand-normalization coefficient (#951f, advisory) (#1011)	2026-06-03 06:28:14 +00:00
bot-backend	69ad6c87fc	feat(forecasting): §9.5 macro coefficient (#951e, advisory) (#1010)	2026-06-03 06:20:04 +00:00
bot-backend	f41e2536d8	feat(forecasting): §9.6 key-rate sensitivity module (#951d, advisory) (#1009)	2026-06-03 06:06:16 +00:00
bot-backend	c7bfc9e22a	feat(forecasting): monthly sales series builder for §9.6 (#951c) (#1008)	2026-06-03 05:52:33 +00:00
bot-backend	ace3b99508	feat(forecasting): monthly macro series + regime classifier (#951b) (#1007)	2026-06-03 05:37:43 +00:00
lekss361	a10592847d	feat(site_finder): future-supply-pressure index (#950 Step 6) (#1006)	2026-06-03 05:10:08 +00:00
Light1YT	900802264a	feat(site-finder): supply_layers v2 compute service (#950 EPIC6 step3+4) (#1004)	2026-06-02 20:34:26 +00:00
Light1YT	59be55f80e	feat(site-finder): per-competitor relevance_weight (#949 PR B) (#1000)	2026-06-02 19:45:38 +00:00
Light1YT	45d61ecff0	feat(site_finder): market-metrics service (#949 PR A, §9.2) (#997)	2026-06-02 19:26:26 +00:00
Light1YT	dbae4b0bda	feat(site-finder): macro_indicator table + backfill + reader (#945 PR A) (#963)	2026-06-02 18:41:28 +00:00
Light1YT	6cb3d7515c	fix(tradein): per-bucket velocity formula + realistic срок продажи (#574) (#631)	2026-05-28 15:35:16 +00:00
lekss361	0567ad2130	feat(22d): domrf_catalog_object scraper — fill ~25 NULL kn_objects cols from SSR __NEXT_DATA__ (#335)	2026-05-17 21:26:22 +00:00
lekss361	689327909c	fix(sf-09): capacity-aware redistribute (round 2) — bounded + correct invariant ... Round 1 (commit bcd7dc8) был broken: на 2-bucket входах surplus уходил в free полностью без учёта capacity → free превышал cap → следующая итерация clamp'ировала его и наоборот. Infinite oscillation в FastAPI handler. Round 2 fix per review BLOCK (#282 comment): - Surplus распределяется пропорционально available capacity (cap - v), не текущему v. Free никогда не вылетит выше cap. - free = строго < cap (не <=) — иначе деление на 0 capacity. - Hard guard `for _ in range(N+1)` — гарантированно завершается. - Pathological (surplus > total_capacity): возвращаем оригинальный pct_map + cap_skipped=True (sum=100 invariant сохранён). - Hamilton round вынесен в _hamilton_round() helper. Tests: - 2-bucket cases (90/10, 70/30, 99/1) expected cap_skipped=True - test_cap_iteration_count_bounded — все pathological завершаются < 100ms - All 13 cases verified standalone (3 fast-path + 7 reproduced + 3 pathological)	2026-05-17 15:30:50 +03:00
lekss361	bcd7dc8f75	fix(sf-09): iterative cap algorithm + cap_skipped flag + unit tests ... Review verdict MINOR (PR #282) — single-pass cap_and_redistribute не гарантировал invariant max ≤ MAX_BUCKET_SHARE_PCT. Surplus от clamped поднимал free выше cap (например {1k:75, studio:15, 2k:10} → studio=39). - backend/app/services/site_finder/best_layouts.py - _cap_and_redistribute → iterative while-loop до сходимости - Returns (result_map, cap_skipped) — флаг для pathological all-clamped - Float work map, Hamilton финальный pass для invariant sum=100 - backend/app/schemas/parcel.py - LayoutTzRecommendation.cap_skipped: bool = False - backend/tests/services/site_finder/test_best_layouts.py - 6 parametrized tests: invariants, 7 failing cases from review, no-dominant unchanged, empty, cap_skipped propagation, normal-case - frontend/src/types/best-layouts.ts — cap_skipped: boolean - frontend/src/components/site-finder/BestLayoutsBlock.tsx - banner условие сменилось с maxPct > 60 на rec.cap_skipped (honest signal)	2026-05-17 15:21:51 +03:00
lekss361	5a4408e0ab	fix(sf-01): time_window honest velocity — inline SQL с реальным фильтром report_month ... Раньше _VELOCITY_DIVISORS делил агрегаты mv_layout_velocity (24 мес) на 4/12 для quarter/year, не меняя реальное окно данных. Теперь inline SQL из objective_corpus_room_month с CAST(:window_interval AS interval). velocity_per_month = deals_window / months_in_window (1.0/3.0/12.0). Разные time_window → разные строки из БД → разный mix/velocity/jk_count. Closes (epic part) #271 item 1	2026-05-17 12:12:19 +03:00
lekss361	8e377059a4	fix(#265 review): CRS 3857→4326 + stable zone_id hash + rename 102 + tx comment ... - #1 (BLOCK): ST_GeogFromGeoJSON → ST_Transform(ST_SetSRID(..., 3857), 4326)::geography so EPSG:3857 features from get_territorial_zones_in_bbox are correctly stored in GEOGRAPHY(POLYGON, 4326) column (matches cad_parcels/cad_buildings pattern) - #2: zone_id fallback changed from counter-based f"{quarter_cad}_{inserted}" to md5(json(props))[:12] — stable between runs, prevents wrong-row ON CONFLICT - #3: rename 101_cad_nspd_overlay_tables.sql → 102_* (101_user_custom_pois.sql exists) - #4: add comment clarifying SQLAlchemy autobegin guarantees outer tx before begin_nested() - Tests: updated test_feature_without_zone_id_uses_fallback to assert md5 format + idempotency	2026-05-17 10:45:52 +03:00
lekss361	f08db43145	feat(#263 sub-A): cad_* overlay schemas + cadastre territorial_zones (ПЗЗ) ... Phase 1 (migration 101): создаёт 5 новых таблиц cad_territorial_zones, cad_red_lines, cad_engineering_structures, cad_risk_zones, cad_opportunity_parcels. Idempotent (IF NOT EXISTS), GIST-индексы. Phase 2 partial (ПЗЗ only): - NSPDBulkClient.get_features_in_bbox_grid — async grid-walk WMS GetFeatureInfo - NSPDBulkClient.get_territorial_zones_in_bbox — convenience wrapper layer 875838 - harvest_quarter Phase 2.5 — вызов get_territorial_zones_in_bbox после grid-walk - _save_territorial_zones — UPSERT loop с SAVEPOINT per-row, ON CONFLICT zone_id 9 unit tests (mock DB): insert/upsert/no-geom/no-id/error-isolation/savepoint pattern.	2026-05-17 10:28:22 +03:00
lekss361	b99ac76463	Merge remote-tracking branch 'forgejo/main' into fix/234-nspd-harvest-eta-badge ... # Conflicts: # backend/app/services/site_finder/quarter_dump_lookup.py	2026-05-17 09:19:12 +03:00
lekss361	990c109756	fix(#234): NSPD harvest ETA badge + SETNX dedupe (Sub-PR C) ... Backend (quarter_dump_lookup.py): - _acquire_harvest_lock: Redis SETNX TTL=120s на quarter, защищает от burst N concurrent analyze, ставящих N одинаковых harvest task в очередь - _trigger_harvest: использует lock перед apply_async, возвращает False если lock уже взят (другой запрос триггернул раньше) - make_empty_result/EMPTY_DUMP_RESULT: новое поле harvest_eta_seconds в nspd_dump dict, типичный harvest_quarter = 60с - /analyze: пробрасывает поле через nspd_dump dict (нет typed schema — response_model=None для /analyze endpoint, dict уходит как есть) Frontend (NspdFreshnessBadge, NspdZoningBlock): - Countdown «НСПД: загрузка ~Nс» вместо бесконечного спиннера - После остановки countdown (remaining=0) NspdZoningBlock показывает «загрузка дольше обычного» + ссылку на ПКК вместо infinite skeleton Tests: 5 новых unit + 2 для empty_result schema (всего +7, pass) Closes #234 (UX-side; data-side resolves когда Sub-PR B + D merged).	2026-05-17 09:13:06 +03:00
lekss361	10b79da2e9	fix(#243): cad_zouit fallback срабатывает когда nspd dump полностью отсутствует (#248)	2026-05-17 06:03:19 +00:00
lekss361	4a48d49474	fix(#242): ekburg_permits SSL — verify=False для CA Минцифры РФ	2026-05-17 08:57:42 +03:00
lekss361	db1ff5b1fc	feat(nspd): denorm nspd_parcels + nspd_buildings tables (#94 PR3 of 4) ... - data/sql/99_nspd_entities_denorm.sql: 2 tables (cad_num PK) + 6 indexes (quarter_cad, GIST geom, partial WHERE purpose ILIKE '%многокв%') - nspd_denorm.py: denorm_parcel_feature / denorm_building_feature / denorm_dump - SAVEPOINT per row via with db.begin_nested() - ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON(:geom), 3857), 4326) - json.dumps(geom, ensure_ascii=False) for psycopg bind - All CAST(:x AS type) — no ❌:type - nspd_sync.py: inline denorm after _upsert_dump in harvest_quarter (non-fatal) - nspd_denorm_backfill.py: Celery one-shot task (per-quarter commit) - admin_etl.py: POST /api/v1/admin/etl/nspd-denorm-backfill (AdminTokenAuth) - 19 tests: coerce helpers, denorm parcel/building, aggregate counts Foundation for downstream features (МКД neighbors lookup, parcel attrs). Part of #94	2026-05-16 20:20:03 +03:00
lekss361	fdb54834e5	fix(nspd): rename migration 89→98, fix red lines ST_Area→ST_Length (#220) ... - data/sql/98_*: rename from 89 to avoid collision with existing 89_drop_dead_brin - _get_red_lines: ST_Length(ST_Intersection(planar)::geography) instead of ST_Area(ST_Intersection(::geography, ::geography)) — fixes PostGIS 3.4 tolerance error and returns correct non-zero length for LINESTRING intersections - Rename intersection_area_sqm → intersection_length_m across schema, TS types, frontend component, and tests; add test_get_red_lines_db_exception_returns_empty Addresses review-bot feedback on PR #220.	2026-05-16 19:33:14 +03:00
lekss361	ed3c128528	feat(nspd): TIER 4 opportunity layers + red lines (#94 PR2 of 4) ... - NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt) + QuarterDump.opportunity field - nspd_sync: harvest_quarter accepts include_opportunity, denorm cols has_auction_parcels + opportunity_count in UPSERT - quarter_dump_lookup: - _get_opportunity_parcels (sort by distance, early-exit on count=0) - _get_red_lines (query existing dump.red_lines core path, layer='red_lines') - SQL 89_*: has_auction_parcels + opportunity_count + partial index - Pydantic: OpportunityParcel + RedLine schemas - /analyze: nspd_opportunity_parcels + nspd_red_lines fields - Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration - Tests: 28 total (11 PR1 + 17 PR2), all pass Red lines uses existing core harvest path (layer 879243 already in dump.red_lines from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed. Part of #94	2026-05-16 19:06:22 +03:00
lekss361	1595157ebf	feat(nspd): enable TIER 3 risk zones harvest + extract in analyze (#94 PR1) ... - harvest_stale_quarters fanout passes include_risks=True to all queued tasks - _get_risk_zones() with ST_Intersects + ST_Area intersection calc - _extract_features_by_layer() generic helper for future PR2+ reuse - RiskZone Pydantic schema; /analyze returns nspd_risk_zones list - NspdRiskZonesBlock component with severity color-coding (red/yellow/orange) - 11 unit tests for extract/intersect/null-area/db-exception paths Part of #94 (TIER 3 risk layers — flooding, landslide, burns, erosion etc)	2026-05-16 16:11:59 +03:00
lekss361	dbf31cc911	feat(etl): objective_complex_mapping backfill via pg_trgm (#203) (#214)	2026-05-16 12:47:09 +00:00
lekss361	adc45f5724	feat(permits): ЕКБ РНС/РВЭ xlsx parser + schema + ETL (#105 PR A) (#211)	2026-05-16 11:41:21 +00:00
lekss361	50547dcbb3	test(deploy): validate Forgejo Actions end-to-end flow (#189)	2026-05-15 21:55:04 +00:00
lekss361	4d492ebc50	perf(cadastre): freshness skip + grid-walk heartbeat + log noise reduction (#187) ... * perf(cadastre): freshness skip + grid-walk heartbeat + reduce WMS log noise Three operational improvements after ekb_full v3 completed at 99.1%: Fix H — per-quarter freshness skip (scrape_cadastre.py): - enqueue_cadastre_harvest queries cad_quarter_stats.fetched_at per quarter - Skip quarters where fetched_at >= NOW() - skip_fresh_hours (default 1h) - extra_config.skip_fresh_hours=0 disables skip (full re-fetch) - Saves ~30k NSPD calls on repeated runs of same scope Fix I — heartbeat in grid-walk loop (bulk_harvest.py:_grid_walk_category): - 225 cells x 70-300ms = 15-75s heartbeat silence - Pass update_progress callback to _grid_walk_category - Call every heartbeat_every (50) cells with phase progress info - Prevents cleanup_zombies false-positive cancellation Fix J — NSPD WMS HTTP 500 noise to debug (bulk_harvest.py:_grid_walk_category): - NSPD WMS endpoint frequently returns HTTP 500 (server-side issue) - Was: logger.warning each, 200+ warnings per quarter - Now: logger.debug (visible via --log-level=DEBUG) 40 tests passing. * fix(cadastre): parse NSPD DD.MM.YYYY dates (Fix K — job 18 root cause) Job 18 (manual_list recovery 11 quarters) crashed every quarter with: psycopg.errors.DatetimeFieldOverflow: date/time field value out of range: '13.03.2023' NSPD returns registration_date in mixed formats: - DD.MM.YYYY for Сооружения / ОНС / ЕНК / ЗОУИТ - YYYY-MM-DD ISO for ЗУ / Здания - YYYY-MM-DD HH:MM:SS with time PG CAST(:date AS date) accepts ISO only — DD.MM.YYYY crashes. Add _parse_nspd_date helper: handles both formats + None/empty. Apply in upsert_construction, upsert_onc, upsert_enk, upsert_zouit, upsert_parcel for registration_date / build_record_registration_date / legal_act_date. 3 new unit tests for the helper (43 tests total). * fix(cadastre): address bot blockers — pivot Fix H to raw_targets + tests Bot review on PR #187 caught: 1. cadastre_jobs has no extra_config column → Fix H crashed on KeyError 2. Missing tests for Fix H + Fix I 3. _parse_nspd_date didn't validate date ranges Changes: - Pivot Fix H: read skip_fresh_hours from raw_targets JSONB (existing column), no schema migration needed - _parse_nspd_date: validate via date.fromisoformat() — rejects 2023-13-45 - Add 3 tests for Fix H in backend/tests/workers/test_scrape_cadastre.py - Add 2 tests for Fix I heartbeat callbacks in test_cadastre_bulk.py - Add 1 test for parse_nspd_date range validation (was: only format) 49 tests passing. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 23:02:37 +03:00
lekss361	5f8df655cc	fix(cadastre): cad_parcels.geom Polygon -> MultiPolygon (migration 93) (#185) ... NSPD returns MultiPolygon for Многоконтурный участок (e.g. 66:41:0105017:4) which crashed upsert_parcel — schema was strict POLYGON. Migration 93: ALTER COLUMN geom Polygon -> MultiPolygon USING ST_Multi(geom), DROP+RECREATE GIST index + back-compat VIEW cad_parcels_geom. upsert_parcel SQL wraps ST_Transform in ST_Multi() — coerces Polygon to MultiPolygon. NSPD returns both types (simple + multi-contour parcels). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 18:52:38 +03:00
lekss361	e3359c4f03	fix(cadastre): filter Point/LineString geometry in upsert_parcel/zouit (#184) ... cad_parcels.geom is geometry(Polygon, 4326) — strict schema. NSPD occasionally returns Point geometry for parcels without detailed boundary, causing INSERT failure: psycopg.errors.InvalidParameterValue: Geometry type (Point) does not match column type (Polygon) This killed pilot v8 at quarter 66:41:0104002 — worker autoretry exhausted, job hung at 25/50 (heartbeat stale). Fix: filter geometry.type at Python level in upsert_parcel + upsert_zouit (same pattern as upsert_quarter_geom_from_feature). Non-Polygon geometry → geom=NULL, raw_props preserved. Tables with permissive GEOMETRY schema (cad_buildings, cad_constructions, cad_enk, cad_oncs) unaffected — they accept any geometry type. Tests: 3 new (Point parcel → geom=NULL, Polygon happy path, LineString zouit). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 18:37:29 +03:00
lekss361	e695bca1e4	fix(cadastre): remove shared phase_state early-exit (race condition) (#183) ... cadastre_jobs.phase_state is shared between ALL parallel Celery workers of the same job. The early-exit triggered as soon as the FIRST worker wrote phase=done via progress_cb — all subsequent workers (reading same shared row) bailed without work. Effect: pilot v7 made 5 NSPD requests for 50 quarters (=0.1/quarter). Full ekb_full job #8: 4950 req / 2408 quarters = 2/quarter — 95% workers early-exited without doing snapshot phase. Root cause of 1.6% parcels coverage. Fix A/B/C only partially helped because most workers never reached Phase 1/1.5. Solution: delete the early-exit. Idempotency is guaranteed by ON CONFLICT DO UPDATE in every upsert_* — re-enqueued tasks write same rows without duplicates. Closes part of #168 (root-cause fix for race condition revealed by #182 metrics). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 17:41:27 +03:00
lekss361	057f8891dc	fix(cadastre): count snapshot+per-cat requests in cadastre_jobs.requests_count (#182) ... scrape_cadastre.py previously incremented requests_count only by grid_walk_requests, so snapshot-only quarters (and the entire ekb_full job after PR #179 skips grid-walk on broken geom) reported 0 NSPD requests — misleading for observability. Add HarvestResult.snapshot_requests counter (Phase 1 search + Phase 1.5 per-cat probes) and a total_requests property. Update Celery task to write total_requests into cadastre_jobs.requests_count. Tests added for the new property + Phase 1 snapshot_requests=1 assertion. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 17:17:01 +03:00
lekss361	905ef443fc	fix(cadastre): auto-heal cad_quarters_geom from CAT_QUARTER_STATS feature (#181) ... 72% of EKB quarters (1735/2408) have broken micro-precision geom in cad_quarters_geom. NSPD returns valid quarter polygon in CAT_QUARTER_STATS (36381) feature but Phase 4 was only consuming stats. Add upsert_quarter_geom_from_feature: CTE-based UPDATE that transforms 3857 polygon to 4326 and writes only when existing geom is NULL or broken (bbox width outside 100-10000m), verified via SQL-side sanity check. Wire into Phase 4 with begin_nested savepoint: malformed NSPD GeoJSON (self-intersecting ring) rollbacks only geom UPDATE, preserves upsert_quarter_stats. Closes part of #168 (follow-up data quality fix for #179). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 17:16:39 +03:00
lekss361	d5b57ccbf1	feat(cadastre): per-category NSPD snapshot — Phase 1.5 (ZOUIT/ENK) (#180) ... Switch base search_by_quarter from thematicSearchId=1 to thematic=1 (includes ZOUIT in features). Add optional category_id param that routes ZOUIT (36940/469039/469040/469042) -> thematicSearchId=5, ENK (39663) -> thematicSearchId=15. Add Phase 1.5 in harvest_quarter: for each PER_CATEGORY_PROBE_CATS category present in meta_counts (meta_total>0), run dedicated per-category search call. Skips if category in GRID_WALK_CATS and bbox is valid (grid-walk gives better coverage). Curl-probe confirmed NSPD API caps at 20/category with no pagination; categoryId param is server-ignored; thematic=1 strictly better than thematicSearchId=1 (includes ZOUIT). Closes part of #168 (ZOUIT/ENK coverage gap). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 16:53:49 +03:00
lekss361	3beb75d68a	fix(cadastre): skip grid-walk for broken quarter geometries (#168) (#179) ... * fix(cadastre): skip grid-walk for broken quarter geometries (#168) Pilot v5 (50 quarters) разоблачил что cad_quarters_geom имеет 1735/2408 (72%) ЕКБ rows с broken micro-precision geom (width ~0.01м instead of 200-4000м). Causes: - grid-walk WMS receives bbox ~1mm × 1mm → 500 ServiceException на all 450 req - One quarter hung 2+ min (silent failure in async loop) Real fix: re-import cad_quarters_geom geometry from NSPD (category 36381). Tracked в follow-up issue. Quick fix (this PR): sanity-check quarter_bbox_3857 — return None если width ИЛИ height not in [100m, 10000m]. Bulk_harvest_quarter then skips grid-walk для broken quarters, только snapshot phase запускается (20 per cat). After this fix: - All 50 pilot quarters получают snapshot (parcels/buildings/etc up to 20 each) - Grid-walk runs only для 673/2408 healthy quarters → no 500 cascades, no hangs * test(cadastre): add quarter_bbox_3857 boundary tests (PR #179 bot blocker) - healthy bbox 750m -> returns tuple - width <100m (broken micro-precision) -> None - width >10000m (oversized) -> None - height <100m (symmetry) -> None - row missing (quarter not in cad_quarters_geom) -> None --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 16:25:06 +03:00
lekss361	99e2210919	fix(cadastre): NSPD response parse — properties.category + top-level meta (#168) (#177) ... * fix(cadastre): NSPD response parse — properties.category + top-level meta (#168) * fix(cadastre): test fixtures use real NSPD shape — properties.category + top-level meta (#168) Bot review of #177 flagged that fixtures still used legacy properties.categoryId + data.meta — meaning CI exercised only fallback branches, not primary paths where the actual bug lived. Pattern of 3 consecutive "live-only" parse fixes (#175 verify, #176 headers, #177 parse) confirmed need for test coverage. Changes: - test_nspd_bulk_client.py: sample_quarter_response → properties.category + meta moved to top-level (real NSPD shape verified live) - test_cadastre_bulk.py: 7 fixtures categoryId → category (regex replace) - test_nspd_bulk_feature_parse_basic: primary path now exercised Plus schema hardening per bot review: - NSPDBulkFeature.category_id: `is not None` check (not truthy `or`) to avoid edge case category=0; int() wrapped in try/except so non-numeric ID (e.g. "ЗУ") doesn't crash upsert_features loop. 35/35 tests pass locally. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 15:40:01 +03:00
lekss361	5da66bbc93	feat(cadastre): bulk_harvest worker + grid-walker + admin API (#168 PR3/5) (#171) ... * feat(cadastre): bulk_harvest_quarter Celery task + grid-walker + saga state (#168 PR3/5) Add bulk cadastre harvest pipeline: - services/cadastre/bulk_harvest.py: async harvest_quarter() orchestrator (4 phases) + 7 upsert helpers (parcels/buildings/constructions/oncs/enks/zouit/quarter_stats) using CAST(:x AS jsonb) pattern, begin_nested() SAVEPOINT per grid-walk upsert - services/cadastre/grid_geometry.py: quarter_bbox_3857 (PostGIS ST_Extent) + generate_grid_click_points (15x15 = 225 sub-bbox grid) - workers/tasks/scrape_cadastre.py: bulk_harvest_quarter_task (acks_late=True, dont_autoretry_for NspdBulkWafError), enqueue_cadastre_harvest, cleanup_zombies - api/v1/admin_cadastre.py: 5 endpoints behind AdminTokenAuth — create/list/get/cancel/resume - Tests: 13 service unit + 8 API tests * fixup(cadastre): drop importorskip (PR2 merged) — imports now top-level (#168 PR3) * fixup(cadastre): tile_size unified param + register slow marker (#168 PR3) Blocker #1: rename tile_width/tile_height → tile_size in generate_grid_click_points. Callers in bulk_harvest.py and test_cadastre_bulk.py used tile_size; def had tile_width+tile_height → TypeError at runtime. Blocker #2 (10 failures in test_admin_cadastre.py) was side-effect of #1: TypeError at import chain (app.main → admin_cadastre → bulk_harvest) broke FastAPI app load → dependency_overrides AttributeError. Now resolved. Also register `slow` pytest marker in pyproject.toml to suppress PytestUnknownMarkWarning. * fixup(cadastre): use importlib.util.find_spec instead of import (#168 PR3) Root cause Blocker #2: `import app.workers.tasks.scrape_cadastre` at module top-level REBOUND `app` from FastAPI instance (line 14) to the Python package. All subsequent `app.dependency_overrides[get_db] = ...` failed with AttributeError because `app` was now the namespace module. Fix: probe module availability with importlib.util.find_spec — does not import or rebind names. FastAPI `app` instance stays intact. All 28 cadastre tests pass locally. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 13:31:32 +03:00