Commit graph

3 commits

Author SHA1 Message Date
5170107405 fix(sql): EMISS PK includes period_type so yearly+Q1 coexist (#1606 follow-up)
Some checks failed
CI / backend-tests (push) Has been cancelled
CI / frontend-tests (push) Has been cancelled
CI / openapi-codegen-check (push) Has been cancelled
CI / changes (push) Has been cancelled
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been cancelled
CI / frontend-tests (pull_request) Has been cancelled
CI / openapi-codegen-check (pull_request) Has been cancelled
Adds period_type TEXT NOT NULL DEFAULT 'unknown' to macro_indicator and
widens the PRIMARY KEY to (indicator_type, region, obs_date, period_type).

Before: yearly aggregate ('год'→obs_date YYYY-01-01) and Q1 ('I квартал'→
same date) shared the same PK slot → ON CONFLICT DO UPDATE made them
overwrite each other despite the in-memory dedup fix in #1687.

After: each EmissRow carries period_type from _emiss_period_granularity
('year'/'quarter'/'month'); DB ON CONFLICT targets include it, so
yearly+Q1 rows genuinely coexist.

Non-EMISS sources (CBR, rosstat open-data, domrf) use period_type='unknown'
(literal in SQL) — they are disambiguated by obs_date already, so the
wider PK is backward-compatible. All ON CONFLICT clauses in
cbr_macro_sync.py and rosstat_macro_sync.py updated accordingly.

Migration: 163_emiss_pk_period_type.sql (idempotent BEGIN/COMMIT).
Tests: 49 passed (emiss + rosstat suite), ruff clean, py_compile OK.
2026-06-17 21:27:34 +03:00
9dbfb03a3d fix(scrapers): disambiguate EMISS yearly vs Q1 in dedup key (#1606)
Some checks failed
CI / changes (push) Successful in 7s
CI / changes (pull_request) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Successful in 1m38s
CI / backend-tests (pull_request) Failing after 8m41s
_YEAR_PERIODS ('год') → month=1 collided with Q1 ('I квартал') → month=1
for the same region/year: both produced obs_date=YYYY-01-01, so the second
series in the XML silently overwrote the first in the in-memory by_key dict.

Fix: add _emiss_period_granularity() and extend the dedup key from
(region, obs_date) to (region, obs_date, granularity). 'year' and 'quarter'
are now distinct slots, so both rows survive the parser and reach the upsert.

Test: test_yearly_and_q1_both_survive_dedup verifies len==2 with both
values present; test_period_granularity covers all classification branches.
2026-06-17 20:39:52 +03:00
1f66dfd400 feat(macro): ЕМИСС среднедушевые доходы (id=57039) -> macro_indicator (#946 part2)
All checks were successful
CI / changes (push) Successful in 6s
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
Deploy / changes (push) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
Deploy / build-backend (push) Successful in 1m31s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m33s
Deploy / deploy (push) Successful in 1m12s
CI / backend-tests (push) Successful in 6m23s
CI / backend-tests (pull_request) Successful in 6m31s
Расширяет Росстат-скрейпер ЕМИСС/fedstat-рядом «Среднедушевые денежные доходы
населения» (fedstat id=57039, OKATO 65=Свердл, квартальный, руб). Добавляет pure
SDMX-ML v1.0 GenericData-парсер (stdlib ElementTree, dev-тестируем на фикстурах) +
EMISS-ветку Celery-таски rosstat_macro_sync (open-data + ЕМИСС, per-source guard,
source='emiss', frequency параметризован, CAST not ::, SAVEPOINT per-row).
income_per_capita проброшен в site_finder/macro (region='sverdl').

ЕМИСС за WAF с dev -> fetch исполняется на проде (verified: httpx POST из
gendesign-worker-1 -> 200 + SDMX, парсер извлёк 9 Свердл-строк). ИПЦ (id=31074) /
индекс цен СМР НЕ landed: многомерные, дефолт-экспорт data.do без Свердл, нужна
dataGrid-выборка (selectedFilterIds + filter-tree AJAX + reCAPTCHA) -- задокументировано.

Refs #946.
2026-06-07 18:44:38 +05:00