feat(macro): construction price index (СМР) via open Rosstat xlsx (#946) #1175

Merged

bot-backend merged 1 commit from feat/macro-smr-946 into main

2026-06-09 06:44:11 +00:00

bot-backend commented

2026-06-09 06:28:57 +00:00

Collaborator

Summary

Lands the construction price index (СМР) — the last open piece of #946 — from the open Rosstat xlsx, NOT the reCAPTCHA-gated fedstat dataGrid.do.

Source: rosstat.gov.ru/storage/mediabank/Invest_ind_stroitel_MM-YYYY.xlsx («Индексы цен производителей на строительную продукцию по РФ»). Parses the monthly «к предыдущему месяцу» section → macro_indicator (indicator_type='construction_price_index', region='rf', monthly, %). Reachable from dev (verify=False, gov-CA — cbr_macro/ekburg_permits precedent).
⚠️ reCAPTCHA NOT bypassed: the captcha-gated dataGrid.do path is deliberately avoided (the open xlsx needs no captcha). EMISS id=31108 documented as a prod-only SDMX alternative (not wired).
Robust URL (the agent's first cut hardcoded the month → would rot): _fetch_construction_latest derives the URL from date.today() with bounded look-back (current month back to 6; first HTTP 200 wins; year-boundary handled; exhaustion raises; 404 breaks fast → no hang).
Wired into rosstat_macro_sync with per-source try/except (a СМР failure can't break demography/income) + SAVEPOINT-per-row. None-not-0 on blank/future months.

Test plan

pytest tests/scrapers/test_rosstat_emiss.py tests/workers/test_rosstat_macro_sync.py → 33 passed (incl. look-back resolution, year-boundary, exhaustion, blank-month skip, annual-section exclusion, graceful empty/garbage/missing-sheet, per-source isolation)
ruff clean; no new dependency (pyproject/uv.lock untouched)
captcha grep → only safety-affirming comments + a test asserting "dataGrid" not in url_template
code-reviewer ✅ APPROVE (4 hard blockers — captcha/hang/fabricated-0/per-source — all clear)
post-deploy smoke (prod): python -m app.services.scrapers.rosstat_emiss confirms the look-back lands a real current file.

Note

No construction-cost channel in macro_coefficient yet (it's demand-side only) → this series is ingested/available for future cost-side use + display, not yet wired to a forecast coefficient (expected for #946 ingest scope).

Refs #946

## Summary Lands the construction price index (**СМР**) — the last open piece of #946 — from the **open Rosstat xlsx**, NOT the reCAPTCHA-gated fedstat `dataGrid.do`. - Source: `rosstat.gov.ru/storage/mediabank/Invest_ind_stroitel_MM-YYYY.xlsx` («Индексы цен производителей на строительную продукцию по РФ»). Parses the monthly «к предыдущему месяцу» section → `macro_indicator` (`indicator_type='construction_price_index'`, region='rf', monthly, %). Reachable from dev (`verify=False`, gov-CA — cbr_macro/ekburg_permits precedent). - **⚠️ reCAPTCHA NOT bypassed**: the captcha-gated `dataGrid.do` path is deliberately avoided (the open xlsx needs no captcha). EMISS id=31108 documented as a prod-only SDMX alternative (not wired). - **Robust URL** (the agent's first cut hardcoded the month → would rot): `_fetch_construction_latest` derives the URL from `date.today()` with bounded look-back (current month back to 6; first HTTP 200 wins; year-boundary handled; exhaustion raises; 404 breaks fast → no hang). - Wired into `rosstat_macro_sync` with **per-source try/except** (a СМР failure can't break demography/income) + SAVEPOINT-per-row. **None-not-0** on blank/future months. ## Test plan - [x] `pytest tests/scrapers/test_rosstat_emiss.py tests/workers/test_rosstat_macro_sync.py` → **33 passed** (incl. look-back resolution, year-boundary, exhaustion, blank-month skip, annual-section exclusion, graceful empty/garbage/missing-sheet, per-source isolation) - [x] `ruff` clean; no new dependency (pyproject/uv.lock untouched) - [x] captcha grep → only safety-affirming comments + a test asserting `"dataGrid" not in url_template` - [x] code-reviewer ✅ APPROVE (4 hard blockers — captcha/hang/fabricated-0/per-source — all clear) - [ ] post-deploy smoke (prod): `python -m app.services.scrapers.rosstat_emiss` confirms the look-back lands a real current file. ## Note No construction-cost channel in `macro_coefficient` yet (it's demand-side only) → this series is ingested/available for future cost-side use + display, not yet wired to a forecast coefficient (expected for #946 ingest scope). Refs #946

bot-backend added 1 commit 2026-06-09 06:28:58 +00:00

feat(macro): land construction price index (СМР) via open Rosstat xlsx (#946 )

CI / changes (push) Successful in 6s

Details

CI / changes (pull_request) Successful in 7s

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / backend-tests (pull_request) Successful in 6m27s

Details

Deploy / build-frontend (push) Has been skipped

Details

CI / frontend-tests (push) Has been skipped

Details

CI / backend-tests (push) Successful in 6m29s

Details

Deploy / changes (push) Successful in 6s

Details

Deploy / build-backend (push) Successful in 1m43s

Details

Deploy / build-worker (push) Successful in 3m3s

Details

Deploy / deploy (push) Successful in 1m15s

Details

f9e045028c

Closes the last #946 gap (СМР). Fetches «Индексы цен производителей на
строительную продукцию по РФ» from the OPEN Rosstat xlsx
(rosstat.gov.ru/storage/mediabank/Invest_ind_stroitel_MM-YYYY.xlsx),
parses the monthly «к предыдущему месяцу» section → macro_indicator
(indicator_type='construction_price_index', region='rf', monthly, %).

The reCAPTCHA-gated fedstat dataGrid.do path is DELIBERATELY NOT used —
the open Rosstat source needs no captcha. EMISS id=31108 documented as a
prod-only SDMX alternative (not wired).

URL is date-derived with bounded look-back (filename changes monthly;
_fetch_construction_latest walks current month back to 6, first 200 wins;
year-boundary handled, exhaustion raises, 404 breaks fast — no hang).
Wired into rosstat_macro_sync with per-source try/except (can't break
demography/income) + SAVEPOINT-per-row. None-not-0 on blank months.
33 tests (incl look-back + year-boundary + graceful), ruff clean.

Note: no construction-cost channel in macro_coefficient yet (demand-side
only) — series is ingested/available for future cost-side use + display.

Refs #946