Commit graph

4 commits

Author SHA1 Message Date
1f66dfd400 feat(macro): ЕМИСС среднедушевые доходы (id=57039) -> macro_indicator (#946 part2)
All checks were successful
CI / changes (push) Successful in 6s
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
Deploy / changes (push) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
Deploy / build-backend (push) Successful in 1m31s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m33s
Deploy / deploy (push) Successful in 1m12s
CI / backend-tests (push) Successful in 6m23s
CI / backend-tests (pull_request) Successful in 6m31s
Расширяет Росстат-скрейпер ЕМИСС/fedstat-рядом «Среднедушевые денежные доходы
населения» (fedstat id=57039, OKATO 65=Свердл, квартальный, руб). Добавляет pure
SDMX-ML v1.0 GenericData-парсер (stdlib ElementTree, dev-тестируем на фикстурах) +
EMISS-ветку Celery-таски rosstat_macro_sync (open-data + ЕМИСС, per-source guard,
source='emiss', frequency параметризован, CAST not ::, SAVEPOINT per-row).
income_per_capita проброшен в site_finder/macro (region='sverdl').

ЕМИСС за WAF с dev -> fetch исполняется на проде (verified: httpx POST из
gendesign-worker-1 -> 200 + SDMX, парсер извлёк 9 Свердл-строк). ИПЦ (id=31074) /
индекс цен СМР НЕ landed: многомерные, дефолт-экспорт data.do без Свердл, нужна
dataGrid-выборка (selectedFilterIds + filter-tree AJAX + reCAPTCHA) -- задокументировано.

Refs #946.
2026-06-07 18:44:38 +05:00
f7c4d7a8c5 feat(macro): Росстат open-data macro scraper -> macro_indicator (#946)
All checks were successful
Deploy / changes (push) Successful in 7s
Deploy / build-backend (push) Successful in 1m50s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m54s
Deploy / deploy (push) Successful in 1m22s
EPIC2 macro-ingest: региональный (Свердловская обл. ОКТМО 65 / ЕКБ) скрейпер
Росстата в унифицированную macro_indicator (м.123). Зеркалит форму cbr_macro:
pure-парсеры + тонкий httpx, Celery-таска с SAVEPOINT-per-row upsert
(CAST not ::, ON CONFLICT по PK indicator_type/region/obs_date).

ЕМИСС/fedstat.ru (SDMX) за WAF (hard-403 на всех путях из dev-IP) -> ИПЦ/доходы/
СМР-цены пока недоступны (документировано в коде). rosstat.gov.ru/opendata
(стандарт 4.0) WAF-free -> приземляет демографию population_total
(sverdl 4.32M / ekb 1.41M, §7.11).

- services/scrapers/rosstat_emiss.py — fetch+parse open-data (registry->meta->data CSV)
- workers/tasks/rosstat_macro_sync.py — Celery upsert (source=rosstat)
- beat: rosstat-macro-sync-monthly; include в celery_app
- site_finder/macro.py: population_total -> region-aware default sverdl (additive)
- tests: 18 offline-тестов парсера + контракта таски

Без миграции (вписано в существующую схему), без новых зависимостей.

Refs #946.
2026-06-07 13:12:01 +00:00
3945d54e3b fix(#945): indicator-aware default region in macro reader + document debt data-quality
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 2m53s
Deploy / build-worker (push) Successful in 3m19s
Deploy / deploy (push) Successful in 1m10s
FOLLOW-UP (Leha: not a bug, follow-up).

get_macro_series / get_latest_macro defaulted region='rf', but CBR mortgage_*
series live under 'sverdl' in macro_indicator (key_rate under 'rf'). Only caller
passes region explicitly so no active leak, but a future caller omitting region
would silently get None — latent footgun. region now str|None; when None →
_canonical_region (mortgage set sourced from _CBR_SERIES_MAP, single source of
truth). Explicit region always wins → existing callers unchanged. Region-binding
tests added. 159 tests pass.

Documented (NOT code-fixed — upstream data-quality): mortgage_debt /
mortgage_overdue stay empty in macro_indicator because cbr_mortgage_series.period
holds value-like garbage (e.g. '10054588.0') for the Debt series — corruption
from the upstream CBR-XLSX scraper that built domrf.db (imported verbatim by
44_import_anton_db.py), NOT an ingest bug. 123_macro_indicator.sql correctly
skips unparseable periods. Needs a separate scraper re-ingest (idempotent
backfill once period is valid). Refs #945
2026-06-04 11:49:41 +05:00
dbae4b0bda feat(site-finder): macro_indicator table + backfill + reader (#945 PR A) (#963)
All checks were successful
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m20s
Deploy / build-backend (push) Successful in 2m32s
Deploy / deploy (push) Successful in 1m14s
Deploy / changes (push) Successful in 6s
2026-06-02 18:41:28 +00:00