Commit graph

13 commits

Author SHA1 Message Date
1883271908 fix(cadastre): NSPD 500-retry + per-layer skip (#252) + disable dead pzz-beat (#259) + geom backfill (#200)
Some checks failed
CI / changes (push) Successful in 7s
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (push) Successful in 6m43s
CI / backend-tests (pull_request) Successful in 6m41s
Deploy / build-backend (push) Blocked by required conditions
Deploy / deploy (push) Blocked by required conditions
Deploy / build-worker (push) Blocked by required conditions
Deploy / build-frontend (push) Blocked by required conditions
Deploy / changes (push) Has been cancelled
#252: NspdBulkServerError на HTTP 5xx + WMS <ServiceException>-body → quarter-level
autoretry; _grid_walk_category флагует layer_failed только если ВСЕ cells 5xx (none ok),
network-errors исключены — пустой/флапающий слой не ложно-скипается. harvest_meta.layer_X_failed.

#259: pzz-sync-monthly beat отключён (PKK6 deprecated, pzz_zones_ekb=0, заменён
zone_regulation_cache+NSPD dumps). logger.exception→warning (стоп GlitchTip BACKEND-1B),
raise сохранён для admin-endpoint.

#200: geom_unavailable флаг (миграция 151) + backfill_parcel_geom task — grid-walk
layer 36368 per quarter лечит 930/964 NULL-geom участков (NSPD search вернул центроиды),
остаток marks unavailable (не ретраить вечно). SAVEPOINT per-cell, idempotent.

Closes #252
Closes #259
Closes #200
2026-06-13 20:45:05 +05:00
f9e045028c feat(macro): land construction price index (СМР) via open Rosstat xlsx (#946)
All checks were successful
CI / changes (push) Successful in 6s
CI / changes (pull_request) Successful in 7s
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (pull_request) Successful in 6m27s
Deploy / build-frontend (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / backend-tests (push) Successful in 6m29s
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m43s
Deploy / build-worker (push) Successful in 3m3s
Deploy / deploy (push) Successful in 1m15s
Closes the last #946 gap (СМР). Fetches «Индексы цен производителей на
строительную продукцию по РФ» from the OPEN Rosstat xlsx
(rosstat.gov.ru/storage/mediabank/Invest_ind_stroitel_MM-YYYY.xlsx),
parses the monthly «к предыдущему месяцу» section → macro_indicator
(indicator_type='construction_price_index', region='rf', monthly, %).

The reCAPTCHA-gated fedstat dataGrid.do path is DELIBERATELY NOT used —
the open Rosstat source needs no captcha. EMISS id=31108 documented as a
prod-only SDMX alternative (not wired).

URL is date-derived with bounded look-back (filename changes monthly;
_fetch_construction_latest walks current month back to 6, first 200 wins;
year-boundary handled, exhaustion raises, 404 breaks fast — no hang).
Wired into rosstat_macro_sync with per-source try/except (can't break
demography/income) + SAVEPOINT-per-row. None-not-0 on blank months.
33 tests (incl look-back + year-boundary + graceful), ruff clean.

Note: no construction-cost channel in macro_coefficient yet (demand-side
only) — series is ingested/available for future cost-side use + display.

Refs #946
2026-06-09 11:28:05 +05:00
25e21c2bff feat(macro): CBR inflation (ИПЦ YoY) -> macro_indicator + activate §9.5 channel (#946)
All checks were successful
CI / frontend-tests (pull_request) Has been skipped
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m45s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m42s
CI / changes (push) Successful in 6s
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
Deploy / deploy (push) Successful in 1m15s
CI / backend-tests (push) Successful in 6m30s
CI / backend-tests (pull_request) Successful in 6m32s
fedstat ИПЦ is reCAPTCHA-blocked; CBR publishes inflation openly. Add
fetch_inflation + parse_inflation_xlsx (CBR UniDbQuery DownloadExcel/132934,
monthly % г/г, region=rf, source=cbr) to cbr_macro.py; upsert
indicator_type=inflation_yoy via the existing cbr_macro_sync task (per-series
guard, SAVEPOINT-per-row, CAST not ::, ON CONFLICT on the PK).

Surface inflation_yoy in MonthlyMacro (frozen, carry-forward) and ACTIVATE the
reserved §9.5 inflation channel (macro_coefficient f_inflation: level-vs-4%-target
nudge, non-positive to avoid double-counting f_rate, excluded from
_RATE_DRIVEN_FACTORS). Channel was DEGRADED (no data) -> now BACKED + consumed;
_CONF_HIGH_MIN_BACKED 4->5. Deterministic (§16/§26); renorm claims the reserved
0.08 slice as designed. Live-verified (2026-04 5.58%); 194 macro + 902 forecasting
tests green. No migration, no new deps.

Refs #946.
2026-06-08 11:41:14 +05:00
e2faab801c fix(sf): pravo.gov66 law_type числовой код =1 Постановление (#1149)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m30s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m21s
Deploy / deploy (push) Successful in 1m15s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-06-07 14:17:52 +00:00
396db20c48 feat(sf): enumerate изъятия/резервирования с pravo.gov66 → land_reservation (#1145)
Some checks failed
Deploy / build-frontend (push) Blocked by required conditions
Deploy / deploy (push) Blocked by required conditions
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Has been cancelled
Deploy / build-worker (push) Has been cancelled
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 13:56:29 +00:00
1f66dfd400 feat(macro): ЕМИСС среднедушевые доходы (id=57039) -> macro_indicator (#946 part2)
All checks were successful
CI / changes (push) Successful in 6s
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
Deploy / changes (push) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
Deploy / build-backend (push) Successful in 1m31s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m33s
Deploy / deploy (push) Successful in 1m12s
CI / backend-tests (push) Successful in 6m23s
CI / backend-tests (pull_request) Successful in 6m31s
Расширяет Росстат-скрейпер ЕМИСС/fedstat-рядом «Среднедушевые денежные доходы
населения» (fedstat id=57039, OKATO 65=Свердл, квартальный, руб). Добавляет pure
SDMX-ML v1.0 GenericData-парсер (stdlib ElementTree, dev-тестируем на фикстурах) +
EMISS-ветку Celery-таски rosstat_macro_sync (open-data + ЕМИСС, per-source guard,
source='emiss', frequency параметризован, CAST not ::, SAVEPOINT per-row).
income_per_capita проброшен в site_finder/macro (region='sverdl').

ЕМИСС за WAF с dev -> fetch исполняется на проде (verified: httpx POST из
gendesign-worker-1 -> 200 + SDMX, парсер извлёк 9 Свердл-строк). ИПЦ (id=31074) /
индекс цен СМР НЕ landed: многомерные, дефолт-экспорт data.do без Свердл, нужна
dataGrid-выборка (selectedFilterIds + filter-tree AJAX + reCAPTCHA) -- задокументировано.

Refs #946.
2026-06-07 18:44:38 +05:00
f7c4d7a8c5 feat(macro): Росстат open-data macro scraper -> macro_indicator (#946)
All checks were successful
Deploy / changes (push) Successful in 7s
Deploy / build-backend (push) Successful in 1m50s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m54s
Deploy / deploy (push) Successful in 1m22s
EPIC2 macro-ingest: региональный (Свердловская обл. ОКТМО 65 / ЕКБ) скрейпер
Росстата в унифицированную macro_indicator (м.123). Зеркалит форму cbr_macro:
pure-парсеры + тонкий httpx, Celery-таска с SAVEPOINT-per-row upsert
(CAST not ::, ON CONFLICT по PK indicator_type/region/obs_date).

ЕМИСС/fedstat.ru (SDMX) за WAF (hard-403 на всех путях из dev-IP) -> ИПЦ/доходы/
СМР-цены пока недоступны (документировано в коде). rosstat.gov.ru/opendata
(стандарт 4.0) WAF-free -> приземляет демографию population_total
(sverdl 4.32M / ekb 1.41M, §7.11).

- services/scrapers/rosstat_emiss.py — fetch+parse open-data (registry->meta->data CSV)
- workers/tasks/rosstat_macro_sync.py — Celery upsert (source=rosstat)
- beat: rosstat-macro-sync-monthly; include в celery_app
- site_finder/macro.py: population_total -> region-aware default sverdl (additive)
- tests: 18 offline-тестов парсера + контракта таски

Без миграции (вписано в существующую схему), без новых зависимостей.

Refs #946.
2026-06-07 13:12:01 +00:00
c78c6ec2d0 feat(site-finder): live CBR key-rate scraper → macro_indicator (#945 PR B) (#964)
All checks were successful
Deploy / build-frontend (push) Has been skipped
Deploy / deploy (push) Successful in 1m10s
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m24s
Deploy / build-worker (push) Successful in 2m53s
2026-06-02 19:05:57 +00:00
lekss361
570c987e78 fix(#231): repair test_rate_limit_semaphore regression + docstring sync
Re-review feedback (PR #260 head 8191a85):

1. tests/scrapers/test_nspd_bulk_client.py:360 — test referenced bulk_mod._SEMAPHORE
   which was renamed to _SEMAPHORE_LIMIT in previous commit. AttributeError на
   полном pytest run. Заменено: assert bulk_mod._SEMAPHORE_LIMIT == 3 (smoke).
   max_concurrent <= 3 уже валидирует throttling — capacity check был лишним.

2. app/services/scrapers/nspd_client.py docstrings (lines 483, 506) — обновлены
   ссылки NSPDBulkClient._SEMAPHORE(3) → NSPDBulkClient._sem (per-instance,
   capacity=3) чтобы соответствовать новой архитектуре.
2026-05-17 10:22:16 +03:00
lekss361
fdced94626 fix(#231): switch search_by_quarter area layers to grid-walk dispatch (Sub-PR B)
- Add _GRID_WALK_LAYERS frozenset (territorial_zones, red_lines,
  engineering_structures, zouit_*, risk_*)
- _fetch_layer dispatch: area layers → get_features_in_bbox_grid(grid_n=7,
  step_m=50) [foundation из PR #247], EGRN layers (parcels, buildings) →
  legacy get_features_in_bbox
- Enrich engineering_structures features с classified_kind через
  classify_engineering_kind (PR #247 classifier)
- Per-layer logger.info heartbeat (method + count) для прогресса
- Bump harvest_quarter soft_time_limit 120 → 600s
  (grid-walk: 11 layers × 49 cells × ~70ms ≈ 40s + retries margin)
- TestFetchLayerDispatch (3 tests): 34 passed, 1 skipped

Pre-push deep-code-reviewer: APPROVE (1 medium nit — opportunity_oopt не в
grid-walk, защищено через include_opportunity=False default; follow-up issue
рекомендован).
2026-05-17 09:46:42 +03:00
4519daf149 feat(#126): Sub-PR A — NSPDClient.get_features_in_bbox_grid + engineering classifier (#247)
All checks were successful
Deploy / build-backend (push) Successful in 1m23s
Deploy / build-frontend (push) Has been skipped
Deploy / deploy (push) Successful in 48s
Deploy / changes (push) Successful in 5s
Deploy / build-worker (push) Successful in 2m18s
2026-05-17 05:54:30 +00:00
lekss361
99e2210919
fix(cadastre): NSPD response parse — properties.category + top-level meta (#168) (#177)
* fix(cadastre): NSPD response parse — properties.category + top-level meta (#168)

* fix(cadastre): test fixtures use real NSPD shape — properties.category + top-level meta (#168)

Bot review of #177 flagged that fixtures still used legacy properties.categoryId
+ data.meta — meaning CI exercised only fallback branches, not primary paths
where the actual bug lived. Pattern of 3 consecutive "live-only" parse fixes
(#175 verify, #176 headers, #177 parse) confirmed need for test coverage.

Changes:
- test_nspd_bulk_client.py: sample_quarter_response → properties.category +
  meta moved to top-level (real NSPD shape verified live)
- test_cadastre_bulk.py: 7 fixtures categoryId → category (regex replace)
- test_nspd_bulk_feature_parse_basic: primary path now exercised

Plus schema hardening per bot review:
- NSPDBulkFeature.category_id: `is not None` check (not truthy `or`)
  to avoid edge case category=0; int() wrapped in try/except so
  non-numeric ID (e.g. "ЗУ") doesn't crash upsert_features loop.

35/35 tests pass locally.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 15:40:01 +03:00
lekss361
b7889d113b
feat(cadastre): NSPDBulkClient + schemas for bulk ingest (#168 PR2/5) (#170)
Add async NSPD client (nspd_bulk_client.py) with 3 new methods:
- search_by_quarter: REST /api/geoportal/v2/search/geoportal — snapshot 20 per cat + meta totals
- wms_feature_info: WMS GetFeatureInfo для grid-walk discovery
- list_objects_in_building: tab-group-data (Q3 deferred — код ready)

Rate limit: asyncio.Semaphore(3) + 0.05s jitter.
429 → exponential backoff 2→4→8s, max 3 retries.
403 (WAF) → raise immediately, no retry.

Pydantic schemas (nspd_bulk.py): NSPDBulkFeature, QuarterSnapshot, ObjectsListing
with extra='allow' to preserve unknown fields → raw_props JSONB downstream.

15 unit tests + 4 e2e (real NSPD, skipped in CI by default).

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 12:50:41 +03:00