8 commits

Author	SHA1	Message	Date
Light1YT	1f66dfd400	feat(macro): ЕМИСС среднедушевые доходы (id=57039) -> macro_indicator (#946 part2) ... Расширяет Росстат-скрейпер ЕМИСС/fedstat-рядом «Среднедушевые денежные доходы населения» (fedstat id=57039, OKATO 65=Свердл, квартальный, руб). Добавляет pure SDMX-ML v1.0 GenericData-парсер (stdlib ElementTree, dev-тестируем на фикстурах) + EMISS-ветку Celery-таски rosstat_macro_sync (open-data + ЕМИСС, per-source guard, source='emiss', frequency параметризован, CAST not ::, SAVEPOINT per-row). income_per_capita проброшен в site_finder/macro (region='sverdl'). ЕМИСС за WAF с dev -> fetch исполняется на проде (verified: httpx POST из gendesign-worker-1 -> 200 + SDMX, парсер извлёк 9 Свердл-строк). ИПЦ (id=31074) / индекс цен СМР НЕ landed: многомерные, дефолт-экспорт data.do без Свердл, нужна dataGrid-выборка (selectedFilterIds + filter-tree AJAX + reCAPTCHA) -- задокументировано. Refs #946.	2026-06-07 18:44:38 +05:00
Light1YT	f7c4d7a8c5	feat(macro): Росстат open-data macro scraper -> macro_indicator (#946) ... EPIC2 macro-ingest: региональный (Свердловская обл. ОКТМО 65 / ЕКБ) скрейпер Росстата в унифицированную macro_indicator (м.123). Зеркалит форму cbr_macro: pure-парсеры + тонкий httpx, Celery-таска с SAVEPOINT-per-row upsert (CAST not ::, ON CONFLICT по PK indicator_type/region/obs_date). ЕМИСС/fedstat.ru (SDMX) за WAF (hard-403 на всех путях из dev-IP) -> ИПЦ/доходы/ СМР-цены пока недоступны (документировано в коде). rosstat.gov.ru/opendata (стандарт 4.0) WAF-free -> приземляет демографию population_total (sverdl 4.32M / ekb 1.41M, §7.11). - services/scrapers/rosstat_emiss.py — fetch+parse open-data (registry->meta->data CSV) - workers/tasks/rosstat_macro_sync.py — Celery upsert (source=rosstat) - beat: rosstat-macro-sync-monthly; include в celery_app - site_finder/macro.py: population_total -> region-aware default sverdl (additive) - tests: 18 offline-тестов парсера + контракта таски Без миграции (вписано в существующую схему), без новых зависимостей. Refs #946.	2026-06-07 13:12:01 +00:00
Light1YT	c78c6ec2d0	feat(site-finder): live CBR key-rate scraper → macro_indicator (#945 PR B) (#964)	2026-06-02 19:05:57 +00:00
lekss361	570c987e78	fix(#231): repair test_rate_limit_semaphore regression + docstring sync ... Re-review feedback (PR #260 head 8191a85): 1. tests/scrapers/test_nspd_bulk_client.py:360 — test referenced bulk_mod._SEMAPHORE which was renamed to _SEMAPHORE_LIMIT in previous commit. AttributeError на полном pytest run. Заменено: assert bulk_mod._SEMAPHORE_LIMIT == 3 (smoke). max_concurrent <= 3 уже валидирует throttling — capacity check был лишним. 2. app/services/scrapers/nspd_client.py docstrings (lines 483, 506) — обновлены ссылки NSPDBulkClient._SEMAPHORE(3) → NSPDBulkClient._sem (per-instance, capacity=3) чтобы соответствовать новой архитектуре.	2026-05-17 10:22:16 +03:00
lekss361	fdced94626	fix(#231): switch search_by_quarter area layers to grid-walk dispatch (Sub-PR B) ... - Add _GRID_WALK_LAYERS frozenset (territorial_zones, red_lines, engineering_structures, zouit_*, risk_*) - _fetch_layer dispatch: area layers → get_features_in_bbox_grid(grid_n=7, step_m=50) [foundation из PR #247], EGRN layers (parcels, buildings) → legacy get_features_in_bbox - Enrich engineering_structures features с classified_kind через classify_engineering_kind (PR #247 classifier) - Per-layer logger.info heartbeat (method + count) для прогресса - Bump harvest_quarter soft_time_limit 120 → 600s (grid-walk: 11 layers × 49 cells × ~70ms ≈ 40s + retries margin) - TestFetchLayerDispatch (3 tests): 34 passed, 1 skipped Pre-push deep-code-reviewer: APPROVE (1 medium nit — opportunity_oopt не в grid-walk, защищено через include_opportunity=False default; follow-up issue рекомендован).	2026-05-17 09:46:42 +03:00
lekss361	4519daf149	feat(#126): Sub-PR A — NSPDClient.get_features_in_bbox_grid + engineering classifier (#247)	2026-05-17 05:54:30 +00:00
lekss361	99e2210919	fix(cadastre): NSPD response parse — properties.category + top-level meta (#168) (#177) ... * fix(cadastre): NSPD response parse — properties.category + top-level meta (#168) * fix(cadastre): test fixtures use real NSPD shape — properties.category + top-level meta (#168) Bot review of #177 flagged that fixtures still used legacy properties.categoryId + data.meta — meaning CI exercised only fallback branches, not primary paths where the actual bug lived. Pattern of 3 consecutive "live-only" parse fixes (#175 verify, #176 headers, #177 parse) confirmed need for test coverage. Changes: - test_nspd_bulk_client.py: sample_quarter_response → properties.category + meta moved to top-level (real NSPD shape verified live) - test_cadastre_bulk.py: 7 fixtures categoryId → category (regex replace) - test_nspd_bulk_feature_parse_basic: primary path now exercised Plus schema hardening per bot review: - NSPDBulkFeature.category_id: `is not None` check (not truthy `or`) to avoid edge case category=0; int() wrapped in try/except so non-numeric ID (e.g. "ЗУ") doesn't crash upsert_features loop. 35/35 tests pass locally. --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 15:40:01 +03:00
lekss361	b7889d113b	feat(cadastre): NSPDBulkClient + schemas for bulk ingest (#168 PR2/5) (#170) ... Add async NSPD client (nspd_bulk_client.py) with 3 new methods: - search_by_quarter: REST /api/geoportal/v2/search/geoportal — snapshot 20 per cat + meta totals - wms_feature_info: WMS GetFeatureInfo для grid-walk discovery - list_objects_in_building: tab-group-data (Q3 deferred — код ready) Rate limit: asyncio.Semaphore(3) + 0.05s jitter. 429 → exponential backoff 2→4→8s, max 3 retries. 403 (WAF) → raise immediately, no retry. Pydantic schemas (nspd_bulk.py): NSPDBulkFeature, QuarterSnapshot, ObjectsListing with extra='allow' to preserve unknown fields → raw_props JSONB downstream. 15 unit tests + 4 e2e (real NSPD, skipped in CI by default). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 12:50:41 +03:00