gendesign/backend/app/services/scrapers
Light1YT f9e045028c
All checks were successful
CI / changes (push) Successful in 6s
CI / changes (pull_request) Successful in 7s
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (pull_request) Successful in 6m27s
Deploy / build-frontend (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / backend-tests (push) Successful in 6m29s
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m43s
Deploy / build-worker (push) Successful in 3m3s
Deploy / deploy (push) Successful in 1m15s
feat(macro): land construction price index (СМР) via open Rosstat xlsx (#946)
Closes the last #946 gap (СМР). Fetches «Индексы цен производителей на
строительную продукцию по РФ» from the OPEN Rosstat xlsx
(rosstat.gov.ru/storage/mediabank/Invest_ind_stroitel_MM-YYYY.xlsx),
parses the monthly «к предыдущему месяцу» section → macro_indicator
(indicator_type='construction_price_index', region='rf', monthly, %).

The reCAPTCHA-gated fedstat dataGrid.do path is DELIBERATELY NOT used —
the open Rosstat source needs no captcha. EMISS id=31108 documented as a
prod-only SDMX alternative (not wired).

URL is date-derived with bounded look-back (filename changes monthly;
_fetch_construction_latest walks current month back to 6, first 200 wins;
year-boundary handled, exhaustion raises, 404 breaks fast — no hang).
Wired into rosstat_macro_sync with per-source try/except (can't break
demography/income) + SAVEPOINT-per-row. None-not-0 on blank months.
33 tests (incl look-back + year-boundary + graceful), ruff clean.

Note: no construction-cost channel in macro_coefficient yet (demand-side
only) — series is ingested/available for future cost-side use + display.

Refs #946
2026-06-09 11:28:05 +05:00
..
__init__.py add interactive parser of наш.дом.рф ЖК+flats with stealth Playwright 2026-04-27 17:58:40 +03:00
cbr_macro.py feat(macro): CBR inflation (ИПЦ YoY) -> macro_indicator + activate §9.5 channel (#946) 2026-06-08 11:41:14 +05:00
documents.py feat(22i): domrf_kn_documents new table + parser (#305) 2026-05-17 15:21:31 +00:00
domrf_catalog.py feat(22d): domrf_catalog.py scraper for per-flat price/status from SSR HTML (#309) 2026-05-17 15:40:01 +00:00
domrf_catalog_object.py fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503) 2026-05-24 10:53:28 +00:00
domrf_kn.py feat(domrf): persist domrf_kn_objects.free_flats (#950 PR A) (#1002) 2026-06-02 20:06:29 +00:00
ekb_geoportal_client.py feat(sf): gknspecial_zone ИРД-harvest → ird_overlays (#1114) 2026-06-07 09:54:53 +00:00
ekb_krt_registry.py feat(sf): scraper реестра КРТ-площадок ЕКБ → ekb_krt_sites (#1126) 2026-06-07 11:24:49 +00:00
ekb_ppt_tep_parser.py feat(sf): ППТ/ПМТ ТЭП-парсер (Табл.11/12/13) → ekb_ppt_tep MVP (#1133) 2026-06-07 11:54:40 +00:00
ekburg_permits.py feat(sf): geocode geom для ekburg_construction_permits из МСК-66 (#1124) 2026-06-07 11:16:45 +00:00
flat_plans.py feat(22c): domrf_kn_flat_plans new table + parser stub (#308) 2026-05-17 15:39:34 +00:00
gisogd_so_client.py feat(sf): ingest функц.зон генплана ЕКБ-2045 из ГИСОГД-СО WFS → ekb_genplan_functional_zone (#1137) 2026-06-07 12:03:39 +00:00
izyatie_client.py feat(sf): OCR-пайплайн изъятия ЕКБ (Tesseract rus) → land_reservation (#1062) 2026-06-07 18:07:06 +03:00
izyatie_ocr.py feat(sf): OCR-пайплайн изъятия ЕКБ (Tesseract rus) → land_reservation (#1062) 2026-06-07 18:07:06 +03:00
nspd_client.py feat(sf): opportunity-ЗУ harvest → ird_overlays (future-supply, #1100) 2026-06-06 19:32:45 +00:00
nspd_denorm.py feat(#126): Sub-PR A — NSPDClient.get_features_in_bbox_grid + engineering classifier (#247) 2026-05-17 05:54:30 +00:00
nspd_lite.py fix(nspd-geo): rosreestr2coord v5 compat — delay arg, tmp cache, dumps default 2026-05-11 14:43:44 +03:00
obj_checks.py feat(scrape-kn): parallel Phase B/C + wire obj_checks + documents + is_ekb auto-derive (#322) 2026-05-17 19:40:16 +00:00
objective.py fix(scrape-objective): stream-parse lots_pf via ijson to avoid OOM (#315) 2026-05-17 16:37:30 +00:00
okn_egrkn_client.py fix(sf): корректное urlencode form-data в okn_egrkn_client (#1147) 2026-06-07 14:13:34 +00:00
page_reservation_parser.py feat(sf): ПАГЕ-парсер изъятия/резервирования → land_reservation (#1118) 2026-06-07 10:15:03 +00:00
pravo_gov66_client.py fix(sf): pravo.gov66 law_type числовой код =1 Постановление (#1149) 2026-06-07 14:17:52 +00:00
rosstat_emiss.py feat(macro): land construction price index (СМР) via open Rosstat xlsx (#946) 2026-06-09 11:28:05 +05:00
stealth.py fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503) 2026-05-24 10:53:28 +00:00