64 commits

Author	SHA1	Message	Date
Light1YT	1f66dfd400	feat(macro): ЕМИСС среднедушевые доходы (id=57039) -> macro_indicator (#946 part2) ... Расширяет Росстат-скрейпер ЕМИСС/fedstat-рядом «Среднедушевые денежные доходы населения» (fedstat id=57039, OKATO 65=Свердл, квартальный, руб). Добавляет pure SDMX-ML v1.0 GenericData-парсер (stdlib ElementTree, dev-тестируем на фикстурах) + EMISS-ветку Celery-таски rosstat_macro_sync (open-data + ЕМИСС, per-source guard, source='emiss', frequency параметризован, CAST not ::, SAVEPOINT per-row). income_per_capita проброшен в site_finder/macro (region='sverdl'). ЕМИСС за WAF с dev -> fetch исполняется на проде (verified: httpx POST из gendesign-worker-1 -> 200 + SDMX, парсер извлёк 9 Свердл-строк). ИПЦ (id=31074) / индекс цен СМР НЕ landed: многомерные, дефолт-экспорт data.do без Свердл, нужна dataGrid-выборка (selectedFilterIds + filter-tree AJAX + reCAPTCHA) -- задокументировано. Refs #946.	2026-06-07 18:44:38 +05:00
Light1YT	f7c4d7a8c5	feat(macro): Росстат open-data macro scraper -> macro_indicator (#946) ... EPIC2 macro-ingest: региональный (Свердловская обл. ОКТМО 65 / ЕКБ) скрейпер Росстата в унифицированную macro_indicator (м.123). Зеркалит форму cbr_macro: pure-парсеры + тонкий httpx, Celery-таска с SAVEPOINT-per-row upsert (CAST not ::, ON CONFLICT по PK indicator_type/region/obs_date). ЕМИСС/fedstat.ru (SDMX) за WAF (hard-403 на всех путях из dev-IP) -> ИПЦ/доходы/ СМР-цены пока недоступны (документировано в коде). rosstat.gov.ru/opendata (стандарт 4.0) WAF-free -> приземляет демографию population_total (sverdl 4.32M / ekb 1.41M, §7.11). - services/scrapers/rosstat_emiss.py — fetch+parse open-data (registry->meta->data CSV) - workers/tasks/rosstat_macro_sync.py — Celery upsert (source=rosstat) - beat: rosstat-macro-sync-monthly; include в celery_app - site_finder/macro.py: population_total -> region-aware default sverdl (additive) - tests: 18 offline-тестов парсера + контракта таски Без миграции (вписано в существующую схему), без новых зависимостей. Refs #946.	2026-06-07 13:12:01 +00:00
lekss361	50fcba1ca0	feat(sf): ingest ОКН-объектов ЕКБ из АИС ЕГРКН (точки+категория) → okn_objects (#1141) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 12:51:18 +00:00
lekss361	189ecc81f8	feat(sf): ingest функц.зон генплана ЕКБ-2045 из ГИСОГД-СО WFS → ekb_genplan_functional_zone (#1137) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 12:03:39 +00:00
lekss361	1e53416277	feat(sf): ППТ/ПМТ ТЭП-парсер (Табл.11/12/13) → ekb_ppt_tep MVP (#1133) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 11:54:40 +00:00
lekss361	57c6d119f5	feat(sf): scraper реестра КРТ-площадок ЕКБ → ekb_krt_sites (#1126) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 11:24:49 +00:00
lekss361	ef15190690	feat(sf): geocode geom для ekburg_construction_permits из МСК-66 (#1124) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 11:16:45 +00:00
lekss361	59f2628e0b	feat(sf): ПАГЕ-парсер изъятия/резервирования → land_reservation (#1118) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 10:15:03 +00:00
lekss361	dcafb32f31	feat(sf): gknspecial_zone ИРД-harvest → ird_overlays (#1114) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 09:54:53 +00:00
lekss361	e7a0af5ddc	fix(sf): ekb_geoportal_client verify=False — росс-гос-CA не в trust-store контейнера (#1103) ... _http_get_json/_http_post_json → verify=False (const _VERIFY_TLS): геопортал ЕКБ отдаёт цепочку с росс-гос-CA, которого нет в trust-store контейнера → verify=True падал CERTIFICATE_VERIFY_FAILED, весь geoportal-слой (C8b/D9b/#1085) не работал бы с прода. Данные публичные open-data, секреты не передаются. Прецедент: nspd_lite/ekburg_permits/pzz_loader. Прод-probe подтвердил. +регрессия-тест. Refs #1067. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 20:10:26 +00:00
lekss361	b1530467f7	feat(sf): opportunity-ЗУ harvest → ird_overlays (future-supply, #1100) ... opportunity_harvest.py — grid-walk НСПД opportunity-слоёв (ЗУ аукцион/схема/свободные/межевание/ККР) по кварталам ЕКБ → ird_overlays с layer_kind='opportunity_*'. Переиспользует движок ird_harvest (#1090): _upsert_feature/_ekb_quarters. +krt_territories=37430 в LAYERS. Beat monthly (1-е 05:30). Self-contained, прод-прогон gated B6. Closes #1086. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 19:32:45 +00:00
lekss361	b4e6e43713	feat(sf): EKB geoportal urbanCard — ПЗЗ-регламент content (#1094) ... Расширение ekb_geoportal_client.py — urbanCard pipeline для ПЗЗ-регламента (#1067 C8): searchByGeom → urbanCard → parse_zone_regulation → ZoneRegulation (zone_index + 3 списка ВРИ 540н + предельные параметры). Метод zone_regulation_at(lon,lat). Supplement к WFS (C7), аддитивно. 11 тестов. Wiring в C8b cache-таблицу. Refs #1067, #1059. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 19:05:47 +00:00
lekss361	4e41f9be73	feat(sf): EKB geoportal WFS client (#1083) ... Новый ekb_geoportal_client.py — WFS-клиент муниципального геопортала ЕКБ (Geometa GeoServer, public). features_at_point (CQL INTERSECTS), features_in_bbox (CQL BBOX), zone_index_at (crosswalk координата→индекс ПЗЗ). EPSG:4326, OGC WFS GetFeature (не WMS — reproject-баг). Единый источник землепользование/КРТ/ППТ/ЗОУИТ для ЕКБ. 7 тестов. Foundation #1067 C7; wiring в C8/D9. Refs #1067. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 18:37:09 +00:00
lekss361	c2e0428803	feat(nspd): get_feature_info FEATURE_COUNT=10 — multi-feature per point (#1080) ... get_feature_info() шлёт WMS FEATURE_COUNT=10 (новый параметр feature_count, default 10). NSPD по умолчанию отдаёт 1 фичу/точку → перекрывающиеся зоны одного слоя теряются (37581 «иные ЗОУИТ» на ЕКБ = 2 фичи). Tile-геометрия не тронута. +2 теста (multi-feature parse, override), существующий обновлён. Refs #1067. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 18:18:07 +00:00
lekss361	38d948f078	fix(nspd): list_layers parse flat layers[] response, not children-tree (#1077) ... NSPDClient.list_layers() парсил _walk_layer_tree по id/children, но НСПД layers-theme-tree отдаёт слои в плоском layers[] с ключом layerId → list_layers() молча возвращал [] для любой темы. Fix: сначала плоский data["layers"], иначе fallback на _walk_layer_tree; defensive None-guard на missing id. +тест на flat-форму, backward-compat сохранён. Refs #1067. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 17:51:14 +00:00
Light1YT	420cda2831	feat(domrf): persist domrf_kn_objects.free_flats (#950 PR A) (#1002)	2026-06-02 20:06:29 +00:00
Light1YT	c78c6ec2d0	feat(site-finder): live CBR key-rate scraper → macro_indicator (#945 PR B) (#964)	2026-06-02 19:05:57 +00:00
Light1YT	4e4762f234	fix(tradein): backfill kn obj_class (parser + 3-pass) (#572) (#628)	2026-05-28 15:04:32 +00:00
Light1YT	609ac8ea9d	fix(tradein): backfill kn price_per_m2 via sale_graph + peer median (#573) (#625)	2026-05-28 13:50:45 +00:00
lekss361	099b8e4b59	fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503)	2026-05-24 10:53:28 +00:00
lekss361	9ffeee7ed2	feat(22d): admin UI buttons (skip/force) + reduce catalog jitter 800→300ms (#352)	2026-05-17 23:38:22 +00:00
lekss361	e5498682ba	fix(22d): commit outer transaction in scrape_catalog_objects orchestrator ... PR #335 использовал `with db.begin_nested():` SAVEPOINT для per-row UPDATE, но outer transaction никогда не commit'ился. После цикла Celery task закрывал session (`db.close()` в finally) → autobegin'd outer tx rollback → все UPDATE'ы теряются. Симптом: task завершается с processed=3 succeeded=3, в Glitchtip нет ошибок, но в БД ни одного catalog_scraped_at != NULL. Fix: добавлен db.commit() после цикла + rollback на ошибку commit'а.	2026-05-18 01:10:06 +03:00
lekss361	0567ad2130	feat(22d): domrf_catalog_object scraper — fill ~25 NULL kn_objects cols from SSR __NEXT_DATA__ (#335)	2026-05-17 21:26:22 +00:00
lekss361	2aa33e0eb8	fix(stealth): bump Semaphore 6->8 for higher concurrency in gather ... Per-object 7 endpoints через asyncio.gather упираются в semaphore limit. 6 -> 8 — ещё +33% concurrency для batch fetch'а.	2026-05-17 23:04:51 +03:00
lekss361	0f289c05b1	fix(scrape-kn): real documents endpoints (5x) + skip obj_checks 404 + sem 3->6 ... PR #322 used placeholder URLs: - /api/object/{id}/documents -> 404 (Playwright showed: 5 real endpoints: document/rpd / developer/report / project/documentation / documentation/other / document/permits). Replaced with 5 parallel fetches + merge into single upsert_documents call. - /api/object/{id}/checks -> 404. 6 checkboxes likely inline in bulk kn/object payload; endpoint /checks does not exist. Commented out _fetch_obj_checks_safe with TODO for separate investigation PR. - BrowserSession Semaphore(3) -> Semaphore(6) — double concurrency for intra-object gather. GlitchTip: kn_scrape_failures run #19 showed 40 failures each for both placeholder endpoints across first 30 objects of 1532.	2026-05-17 22:59:44 +03:00
lekss361	f2983c86d7	feat(scrape-kn): parallel Phase B/C + wire obj_checks + documents + is_ekb auto-derive (#322)	2026-05-17 19:40:16 +00:00
lekss361	389e141ec9	fix(scrape-objective): stream-parse lots_pf via ijson to avoid OOM (#315)	2026-05-17 16:37:30 +00:00
lekss361	4bed58b43c	feat(22d): domrf_catalog.py scraper for per-flat price/status from SSR HTML (#309)	2026-05-17 15:40:01 +00:00
lekss361	b37e011aaa	feat(22c): domrf_kn_flat_plans new table + parser stub (#308)	2026-05-17 15:39:34 +00:00
lekss361	6612c1cd2b	feat(22begh): schema extension — flats+6 / objects+30 / developers+4 / metro_top3 (#306)	2026-05-17 15:26:59 +00:00
lekss361	171a029997	feat(22i): domrf_kn_documents new table + parser (#305)	2026-05-17 15:21:31 +00:00
lekss361	3183abe41b	fix(22j): DEDUP domrf_kn_infrastructure + ON CONFLICT (issue #297) (#304)	2026-05-17 15:15:03 +00:00
lekss361	0b12a55eb6	fix(bug-22a): map missing kn-API fields (flat_number/living_area/wall_type/energy_eff) + backfill (#302)	2026-05-17 14:59:02 +00:00
lekss361	bfe16ae71b	fix(sf-16): derive price_per_m2 from price_rub in _norm_flat + backfill SQL (#298)	2026-05-17 14:32:23 +00:00
lekss361	3daf2a1fb1	fix(sf-15): extract obj_class from aiDescription (DOM.RF list API never returns objClass field) ... Add _extract_obj_class_from_ai() with regex patterns for Комфорт/Бизнес/Премиум/Элит/Стандарт. _norm_object() uses it as fallback; objClass from API still takes priority if ever returned. Add backfill SQL 105 to retroactively populate existing rows from raw JSON payloads. Coverage: ~1806/4548 rows will be filled; remainder NULL is expected (Нежилое/no AI desc).	2026-05-17 17:02:07 +03:00
lekss361	570c987e78	fix(#231): repair test_rate_limit_semaphore regression + docstring sync ... Re-review feedback (PR #260 head 8191a85): 1. tests/scrapers/test_nspd_bulk_client.py:360 — test referenced bulk_mod._SEMAPHORE which was renamed to _SEMAPHORE_LIMIT in previous commit. AttributeError на полном pytest run. Заменено: assert bulk_mod._SEMAPHORE_LIMIT == 3 (smoke). max_concurrent <= 3 уже валидирует throttling — capacity check был лишним. 2. app/services/scrapers/nspd_client.py docstrings (lines 483, 506) — обновлены ссылки NSPDBulkClient._SEMAPHORE(3) → NSPDBulkClient._sem (per-instance, capacity=3) чтобы соответствовать новой архитектуре.	2026-05-17 10:22:16 +03:00
lekss361	fdced94626	fix(#231): switch search_by_quarter area layers to grid-walk dispatch (Sub-PR B) ... - Add _GRID_WALK_LAYERS frozenset (territorial_zones, red_lines, engineering_structures, zouit_*, risk_*) - _fetch_layer dispatch: area layers → get_features_in_bbox_grid(grid_n=7, step_m=50) [foundation из PR #247], EGRN layers (parcels, buildings) → legacy get_features_in_bbox - Enrich engineering_structures features с classified_kind через classify_engineering_kind (PR #247 classifier) - Per-layer logger.info heartbeat (method + count) для прогресса - Bump harvest_quarter soft_time_limit 120 → 600s (grid-walk: 11 layers × 49 cells × ~70ms ≈ 40s + retries margin) - TestFetchLayerDispatch (3 tests): 34 passed, 1 skipped Pre-push deep-code-reviewer: APPROVE (1 medium nit — opportunity_oopt не в grid-walk, защищено через include_opportunity=False default; follow-up issue рекомендован).	2026-05-17 09:46:42 +03:00
lekss361	4a48d49474	fix(#242): ekburg_permits SSL — verify=False для CA Минцифры РФ	2026-05-17 08:57:42 +03:00
lekss361	4519daf149	feat(#126): Sub-PR A — NSPDClient.get_features_in_bbox_grid + engineering classifier (#247)	2026-05-17 05:54:30 +00:00
lekss361	db1ff5b1fc	feat(nspd): denorm nspd_parcels + nspd_buildings tables (#94 PR3 of 4) ... - data/sql/99_nspd_entities_denorm.sql: 2 tables (cad_num PK) + 6 indexes (quarter_cad, GIST geom, partial WHERE purpose ILIKE '%многокв%') - nspd_denorm.py: denorm_parcel_feature / denorm_building_feature / denorm_dump - SAVEPOINT per row via with db.begin_nested() - ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON(:geom), 3857), 4326) - json.dumps(geom, ensure_ascii=False) for psycopg bind - All CAST(:x AS type) — no ❌:type - nspd_sync.py: inline denorm after _upsert_dump in harvest_quarter (non-fatal) - nspd_denorm_backfill.py: Celery one-shot task (per-quarter commit) - admin_etl.py: POST /api/v1/admin/etl/nspd-denorm-backfill (AdminTokenAuth) - 19 tests: coerce helpers, denorm parcel/building, aggregate counts Foundation for downstream features (МКД neighbors lookup, parcel attrs). Part of #94	2026-05-16 20:20:03 +03:00
lekss361	ed3c128528	feat(nspd): TIER 4 opportunity layers + red lines (#94 PR2 of 4) ... - NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt) + QuarterDump.opportunity field - nspd_sync: harvest_quarter accepts include_opportunity, denorm cols has_auction_parcels + opportunity_count in UPSERT - quarter_dump_lookup: - _get_opportunity_parcels (sort by distance, early-exit on count=0) - _get_red_lines (query existing dump.red_lines core path, layer='red_lines') - SQL 89_*: has_auction_parcels + opportunity_count + partial index - Pydantic: OpportunityParcel + RedLine schemas - /analyze: nspd_opportunity_parcels + nspd_red_lines fields - Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration - Tests: 28 total (11 PR1 + 17 PR2), all pass Red lines uses existing core harvest path (layer 879243 already in dump.red_lines from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed. Part of #94	2026-05-16 19:06:22 +03:00
lekss361	adc45f5724	feat(permits): ЕКБ РНС/РВЭ xlsx parser + schema + ETL (#105 PR A) (#211)	2026-05-16 11:41:21 +00:00
lekss361	9ee0b07003	feat(scrapers): search_by_quarter orchestrator + QuarterDump (#94 pt.2/4) (#109) ... Sprint 1.1 item #1 из плана #94 part 2. Foundation для PKK harvest pipeline — 1 vacuum (search) + N layer fetches → comprehensive snapshot всех NSPD данных в пределах квартала. Базис для G1 #28 ПЗЗ, G3 #30 ЗОУИТ, P2 #46 neighbors, E1 #51 parcels backfill, #96 ЕГРН помещения. Backend (nspd_client.py): - New QuarterDump frozen dataclass (slots=True): quarter + per-layer feature lists (parcels/buildings/territorial_zones/red_lines/engineering + zouit dict + risks dict) + bbox_3857 + layers_fetched (immutable tuple) + fetched_at_utc + total_features property. - New NSPDClient.search_by_quarter(quarter_cad, include_zouit=True, include_risks=False): search → bbox → bulk fetch per layer phase. Cost 6/11/22 requests. - New _geojson_bbox_3857() module-level helper — recursive coord walker. - Class constants QUARTER_CORE_LAYERS / QUARTER_ZOUIT_LAYERS / QUARTER_RISK_LAYERS. Empty-quarter (NSPD не нашёл cad): quarter=None, bbox=None, all lists empty, zouit/risks dicts populated с пустыми lists (структурно стабильно), layers_fetched=('search',). Tests: +12 tests (31 total, no network). Code review (code-reviewer pre-push): MINOR, fixed 3 of 5: - datetime import → module-level - layers_fetched → tuple[str, ...] (immutable in frozen dataclass) - docstring clarified empty-quarter semantics Bonus: ruff UP038 isinstance tuple → union syntax. Part of #94. Sprint 1.1: 4 PRs total. Next: migration → Celery → integration. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-12 18:11:30 +03:00
lekss361	bc092b58f1	feat(scrapers): NSPD client foundation (#94) — search + WMS + layers (#98) ... * feat(scrapers): NSPD client foundation (#94) — search_by_cad + WMS + layers Foundation для G1 #28 ПЗЗ, G2 #29 ВРИ, G3 #30 ЗОУИТ, E1 #51 cad_parcels, I3 #44 engineering, и поддержки on-demand #93. Backend (`app/services/scrapers/nspd_client.py`): - `NSPDClient` с 4 core методами: - `search_by_cad(cad, thematic_id)` — REST /api/geoportal/v2/search возвращает GeoJSON + ВРИ + land_category + cost_value за 1 запрос - `get_feature_info(layer_id, lon, lat, buffer_m=100)` — WMS GetFeatureInfo на точке: какие feature'ы layer'а покрывают (lon,lat) - `get_features_in_bbox(layer_id, bbox_3857)` — bulk через WMS workaround (WFS GetCapabilities → 404, поэтому большой bbox + центральная I/J точка) - `list_layers(theme_id)` — каталог слоёв в теме (PKK=1, ARN=665) - Typed responses: NSPDFeature, NSPDSearchResult, NSPDLayer (frozen dataclasses) - LAYERS catalog: 32 layer-id с семантическими именами (territorial_zones=875838, zouit_engineering=37578, risk_flooding=872205, etc) — TIER 1-6 per #94 - Coordinate helpers: lonlat_to_3857(), bbox_around_point_m() - Reuse: HEADERS + SSL ctx + fetch_geoportal из existing nspd_lite (WAF-compatible urllib trick). Rate limit через rate_ms. - WAF/Rate-limit: raises NspdLiteWafError на 403/429 — caller backoff. Tests (`backend/tests/test_nspd_client.py`): 14 unit tests, no network. - LAYERS catalog sanity (territorial_zones=875838 закрывает G1) - Coordinate transforms (zero, ЕКБ center, bbox) - NSPDFeature.from_raw parsing (full / missing fields) - NSPDSearchResult helpers (empty/first) - _walk_layer_tree (flat / nested / empty) - search_by_cad через monkeypatch fetch_geoportal Также включён micro-fix из PR #95 review (cosmetic): - parcels.py inline comment "до 25s" → "до 15s" (matched _INLINE_FETCH_WAIT_S) Scope NOT в этом PR (отдельные follow-up issues, sequential rule): - Schema migrations (cad_parcels_geom +columns, nspd_territorial_zones, nspd_zouit, nspd_red_lines, nspd_risk_zones tables) → отдельный PR - Celery tasks (sync_territorial_zones_bbox, sync_zouit_*, sync_risk_*) → отдельный PR использует client - Admin endpoints (trigger / status) → отдельный PR - Замена on-demand fetch на nspd_client → PR after schema ready Closes part of #94 (foundation only — sub-issues for schema/tasks). * fix(scrapers): address PR #98 auto-review minor feedback Backend (nspd_client.py): 1. (line 100+) bbox_around_point_m signature split на multi-line. 2. (lazy imports) import json, time → module-level. 3. (duplicate SSL ctx) Reuse _SSL_CTX из nspd_lite через explicit import. 4. (shape defense) list_layers чекает {"data": [...]} wrapper + non-dict/list garbage → warning + empty. 5. (cad_num docstring) search_by_cad документирует формат + ссылку на validate_cad_format. Tests: +5 mock тестов (19/19 passed): - get_feature_info URL builder + parsing - get_features_in_bbox bbox propagation - list_layers_walks_tree_response - list_layers_handles_data_wrapper - list_layers_handles_garbage_response Backend (parcels.py): docstring '~25с' → '~15с'. Per auto-review on be14388. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-12 09:47:45 +03:00
lekss361	ce5e29f92e	fix(nspd-geo): rosreestr2coord v5 compat — delay arg, tmp cache, dumps default ... Three incremental bugs surfaced while wiring rosreestr2coord v5 into our worker. All in backend/app/services/scrapers/nspd_lite.py. 1. `Area.__init__()` removed `delay` kwarg in v5 (was throttling). Drop it from our call. Rate limit now happens in worker via time.sleep(rate_ms/1000) after each fetch. 2. `Area(..., use_cache=True)` default writes media cache to ./tmp/ relative to CWD. In our docker image CWD=/app (owned by root), worker runs as non-root → PermissionError on every target. Fix: use_cache=False, media_path="/tmp/rosreestr2coord". 3. `Area.to_geojson_poly(dumps=True)` default returns JSON-serialized STRING, not dict (changed in v5). Worker's _persist_target expects dict with .get("properties") → AttributeError. Fix: dumps=False. Vault entries: - fixes/Bug_Rosreestr2coord_Delay_Arg_v5_Fixed.md - fixes/Bug_Nspd_Geo_Tmp_Permission_Fixed.md - fixes/Bug_Nspd_Geo_Str_Object_No_Get_Fixed.md	2026-05-11 14:43:44 +03:00
lekss361	b19ca1ed41	fix(nspd-geo): rosreestr2coord cache permission in non-root worker ... rosreestr2coord.Area() by default creates ./tmp/ media cache relative to CWD. In our worker container CWD=/app (owned by root), runs as non-root user `app` → PermissionError: '/app/tmp' on every target. Fix: pass explicit kwargs to Area(): - use_cache=False (we don't need cache — each cad_num is unique) - media_path='/tmp/rosreestr2coord' (world-writable in container) Vault: fixes/Bug_Nspd_Geo_Tmp_Permission_Fixed.md	2026-05-11 14:22:11 +03:00
lekss361	3919a40c49	refactor(nspd-geo): unify on rosreestr2coord v5, drop legacy column ... Объединяю несколько связанных изменений вокруг NSPD geo bulk-fetcher: Adapt to rosreestr2coord v5 API: - nspd_lite.fetch_via_rosreestr2coord: drop `delay` kwarg from Area() (removed upstream in v5); keep it in our function signature for backward-compat, comment why. - nspd_geo worker: add explicit time.sleep(rate_ms/1000) after lib-branch fetch — in v4 the library throttled internally via delay, in v5 rate limiting is the caller's job. Без этого получали Area.__init__() unexpected kwarg `delay` на каждом target. Drop use_rosreestr2coord switch: - Removed urllib-vs-lib choice everywhere. We always use community rosreestr2coord library — авторы регулярно обновляют WAF-tricks, наш urllib-fetcher (fetch_geoportal) уже неактуален. - admin_scrape.py: Pydantic schema, INSERT, SELECT, API response cleaned of `use_rosreestr2coord`. - nspd_geo.enqueue_geo_job: param dropped, INSERT shrunk. - worker process loop: dropped `if use_lib:` branch + import of fetch_geoportal. - frontend/geo/page.tsx: removed checkbox + GeoJob.use_rosreestr2coord field + POST body field. DB column drop: - data/sql/78_drop_use_rosreestr2coord.sql (NEW): DROP COLUMN nspd_geo_jobs.use_rosreestr2coord + CREATE OR REPLACE VIEW v_scrape_runs_unified (которая depended on the column). - data/sql/77_nspd_geo_jobs.sql: cleaned historical DDL for fresh setups. - Migration applied to prod (in-conversation via postgres MCP). Frontend polish: - Thematic ID changed from free-form number input to labeled select (1=parcel / 2=quarter / 4=admin / 5=building / 7=zone / 15=complex). - Auto-sync thematic_id from Job kind on change (override possible). - ScrapeLogsPanel: extended union type with "nspd_geo" + fixed /admin/scrape/geo to pass scraperType="nspd_geo" (was "nspd", filtering empty legacy nspd_scrape_log table; real logs live in nspd_geo_log via v_scrape_log_unified). Verified: ruff ✓, tsc --noEmit ✓, migration ran (BEGIN..COMMIT clean). Deploy order safe: prod column уже удалена → новый backend код, который не INSERT'ит use_rosreestr2coord, совпадёт со схемой после deploy.	2026-05-11 14:13:33 +03:00
lekss361	79e2e94e09	refactor(nspd): remove Playwright-based scraper, rosreestr2coord = default ... УДАЛЕНО (~1700 строк): * backend/app/services/scrapers/nspd_kn.py (Playwright + WAF-bypass) * backend/app/workers/tasks/scrape_nspd.py * frontend/src/app/admin/scrape/nspd/page.tsx * 5 NSPD admin endpoints (POST /nspd, /nspd/release-lock, GET /nspd/runs, /logs, /coverage) * NSPD beat schedule + worker_ready resume hook + nav tab + link * _nspd_default_regions() helper СОХРАНЕНО (история): * nspd_scrape_runs / nspd_scrape_log таблицы (3 row + log) для аудита * UNION ALL в v_scrape_runs_unified / v_scrape_log_unified — старые runs видны под фильтром scraper_type='nspd' * settings scrape_nspd_* помечены DEPRECATED CHANGED: * nspd_geo_jobs.use_rosreestr2coord DEFAULT FALSE → TRUE * UI checkbox "rosreestr2coord lib" по умолчанию = checked * /admin/scrape/all + /admin/scrape/geo — единственный путь к NSPD-данным NSPD-доступ теперь только через rosreestr2coord (community lib) — upstream поддерживает обновления headers/WAF-tricks. Локальный nspd_lite.py (urllib) остаётся как fallback (use_rosreestr2coord=FALSE).	2026-05-11 09:21:12 +03:00
lekss361	54dbc77348	feat(geo): NSPD bulk-fetcher без Playwright + resume-friendly UI ... ОТКРЫТИЕ: stdlib urllib проходит WAF nspd.gov.ru (TLS-fingerprint stdlib отличается от mainstream HTTP-clients). Это позволяет полностью убрать Playwright + Chromium для NSPD-задач. * nspd_lite.py: urllib + ssl._create_unverified_context(). 4 публичных fetcher'а (geoportal/quarter/parcel/building) + fetch_via_rosreestr2coord fallback на community-lib * schema 77: nspd_geo_jobs (журнал) + nspd_geo_targets (cad-номера со статусом pending/done/failed). Resume-state в БД. * tasks/nspd_geo.py: Celery task process_nspd_geo_job(job_id). Heartbeat каждые 5 items, WAF backoff 30s × 2^N (max 8 → paused). UPSERT в cad_quarters_geom / cad_buildings → идемпотентно. * worker_ready hook: stale jobs (>10min без heartbeat) автоматически re-enqueue после redeploy. Никаких потерь прогресса. * 4 admin endpoint + UI /admin/scrape/geo с формой запуска (manual_list / rosreestr_pending для авто-наполнения из ДДУ-кварталов), progress-bars и cancel/resume кнопками per job. * settings: use_nspd_lite=True, nspd_lite_rate_ms=600. * dep: rosreestr2coord>=4.0.0 (community lib для fallback). TODO следующих шагов: - smoke-test на проде через SSH (validation RU-IP не банит urllib) - feature toggle USE_NSPD_LITE в scrape_nspd.py для переключения с старого Playwright-пути - docker lean image (-300 MB после убирания Chromium) - расширение SCRAPE_NSPD_DEFAULT_REGIONS на 66,74,72,59 (Челябинск/Тюмень/Пермь)	2026-05-11 08:53:28 +03:00
lekss361	aaa5d44e77	feat(objective): integrate api.objctv.ru — auth, schema, weekly sync ... backend/app/services/scrapers/objective.py: ObjectiveClient — GetToken (Bearer cached в Redis 25 min TTL) + GetReport v2 с UseDdu/UseDkp. 4 высокоуровневых метода: report_corpuses_summary, report_lots_summary, report_corpuses_per_flat, report_lots_per_flat. Retry 401/429/5xx, rate-limit 500ms, brotli. backend/app/workers/tasks/scrape_objective.py: Celery task sync_objective_group — еженедельно тянет 4 канон. отчёта по группе, raw payload в objective_raw_reports. celery_app.py: +include + beat «0 5 * * mon». data/sql/68_schema_objective.sql: 6 таблиц — runs, raw_reports (jsonb), oks (готовые ТЭП + escrow/debt), lots, lots_history (per-flat per-day), complex_mapping (Objective ↔ domrf_kn_objects, is_reviewed workflow). data/sql/69_objective_smoke.py: stand-alone GetToken + 4 отчёта → data/raw/objective_smoke/<ts>/. Используется один раз чтобы понять реальную схему payload перед написанием parser-слоя. config: OBJECTIVE_API_KEY, OBJECTIVE_DEFAULT_GROUP=Екатеринбург, OBJECTIVE_SYNC_CRON='0 5 * * mon'.	2026-05-07 21:04:00 +03:00