Commit graph

72 commits

Author SHA1 Message Date
a49ee2ccfc fix(domrf_kn): детерминированный sha256-fallback для flat_id (#1208)
All checks were successful
CI / changes (push) Successful in 6s
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (push) Successful in 6m27s
CI / backend-tests (pull_request) Successful in 6m27s
Fallback flat_id в _norm_flat использовал abs(hash(elem)) % (2**63 - 1).
В CPython hash(str) РАНДОМИЗИРОВАН per-process (PYTHONHASHSEED нигде в
репо не зафиксирован — uvicorn/celery не выставляют его). Эффект:

- При resume упавшего sweep (resume_kn_run в новом процессе воркера) до
  10 объектов после checkpoint перечитываются. Квартиры без flatId
  получают ДРУГОЙ hash-id → ON CONFLICT (id, snapshot_date) не
  срабатывает → дубли строк одной квартиры в одном snapshot.
- То же при повторном прогоне за ту же дату — каждый раз новый id.
- Дубли инфлируют все агрегации (units_sold/price медианы, supply_layers).

Patch:
- abs(hash(elem)) → int.from_bytes(sha256(elem)[:8], "big") % (2**63 - 1).
- sha256 стабилен между процессами/перезапусками. 8 байт → BIGINT-fit.

6 новых юнит-тестов (test_domrf_kn_normalize.py): formula matches sha256,
stable across calls, distinct elems→distinct ids, flatId wins over
fallback, no-id→None, BIGINT-fit. ruff clean.

Closes #1208
2026-06-13 11:01:41 +05:00
f9e045028c feat(macro): land construction price index (СМР) via open Rosstat xlsx (#946)
All checks were successful
CI / changes (push) Successful in 6s
CI / changes (pull_request) Successful in 7s
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (pull_request) Successful in 6m27s
Deploy / build-frontend (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / backend-tests (push) Successful in 6m29s
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m43s
Deploy / build-worker (push) Successful in 3m3s
Deploy / deploy (push) Successful in 1m15s
Closes the last #946 gap (СМР). Fetches «Индексы цен производителей на
строительную продукцию по РФ» from the OPEN Rosstat xlsx
(rosstat.gov.ru/storage/mediabank/Invest_ind_stroitel_MM-YYYY.xlsx),
parses the monthly «к предыдущему месяцу» section → macro_indicator
(indicator_type='construction_price_index', region='rf', monthly, %).

The reCAPTCHA-gated fedstat dataGrid.do path is DELIBERATELY NOT used —
the open Rosstat source needs no captcha. EMISS id=31108 documented as a
prod-only SDMX alternative (not wired).

URL is date-derived with bounded look-back (filename changes monthly;
_fetch_construction_latest walks current month back to 6, first 200 wins;
year-boundary handled, exhaustion raises, 404 breaks fast — no hang).
Wired into rosstat_macro_sync with per-source try/except (can't break
demography/income) + SAVEPOINT-per-row. None-not-0 on blank months.
33 tests (incl look-back + year-boundary + graceful), ruff clean.

Note: no construction-cost channel in macro_coefficient yet (demand-side
only) — series is ingested/available for future cost-side use + display.

Refs #946
2026-06-09 11:28:05 +05:00
25e21c2bff feat(macro): CBR inflation (ИПЦ YoY) -> macro_indicator + activate §9.5 channel (#946)
All checks were successful
CI / frontend-tests (pull_request) Has been skipped
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m45s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m42s
CI / changes (push) Successful in 6s
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
Deploy / deploy (push) Successful in 1m15s
CI / backend-tests (push) Successful in 6m30s
CI / backend-tests (pull_request) Successful in 6m32s
fedstat ИПЦ is reCAPTCHA-blocked; CBR publishes inflation openly. Add
fetch_inflation + parse_inflation_xlsx (CBR UniDbQuery DownloadExcel/132934,
monthly % г/г, region=rf, source=cbr) to cbr_macro.py; upsert
indicator_type=inflation_yoy via the existing cbr_macro_sync task (per-series
guard, SAVEPOINT-per-row, CAST not ::, ON CONFLICT on the PK).

Surface inflation_yoy in MonthlyMacro (frozen, carry-forward) and ACTIVATE the
reserved §9.5 inflation channel (macro_coefficient f_inflation: level-vs-4%-target
nudge, non-positive to avoid double-counting f_rate, excluded from
_RATE_DRIVEN_FACTORS). Channel was DEGRADED (no data) -> now BACKED + consumed;
_CONF_HIGH_MIN_BACKED 4->5. Deterministic (§16/§26); renorm claims the reserved
0.08 slice as designed. Live-verified (2026-04 5.58%); 194 macro + 902 forecasting
tests green. No migration, no new deps.

Refs #946.
2026-06-08 11:41:14 +05:00
5da4271235 feat(sf): OCR-пайплайн изъятия ЕКБ (Tesseract rus) → land_reservation (#1062)
All checks were successful
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / changes (push) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (push) Successful in 6m22s
CI / backend-tests (pull_request) Successful in 6m19s
- Dockerfile: добавлены tesseract-ocr tesseract-ocr-rus в runner apt-слой
- pyproject.toml: pymupdf>=1.24 (рендер PDF без poppler) + pytesseract>=0.3.13
- izyatie_client.py: list_izyatie_documents() + fetch_pdf() с екатеринбург.рф
- izyatie_ocr.py: ocr_pdf_text() (PyMuPDF→PIL→Tesseract rus) + extract_izyatie_records()
  с нормализацией OCR-шума (пробелы в кад-номерах, кирилл. О→0, б→6)
- izyatie_ocr_ingest.py: Celery task → land_reservation UPSERT (SAVEPOINT per-row)
- beat_schedule.py: izyatie-ocr-ingest-weekly (пятница 07:00 МСК)
- 33 теста, все зелёные; без реального Tesseract/сети в CI
2026-06-07 18:07:06 +03:00
e2faab801c fix(sf): pravo.gov66 law_type числовой код =1 Постановление (#1149)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m30s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m21s
Deploy / deploy (push) Successful in 1m15s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-06-07 14:17:52 +00:00
a6e100b52a fix(sf): корректное urlencode form-data в okn_egrkn_client (#1147)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m29s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m39s
Deploy / deploy (push) Successful in 1m14s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-06-07 14:13:34 +00:00
9ee35bebf0 fix(sf): User-Agent header for okn-mk searchMap (bypass 403) (#1146)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m14s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m30s
Deploy / deploy (push) Successful in 1m10s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-06-07 13:57:34 +00:00
396db20c48 feat(sf): enumerate изъятия/резервирования с pravo.gov66 → land_reservation (#1145)
Some checks failed
Deploy / build-frontend (push) Blocked by required conditions
Deploy / deploy (push) Blocked by required conditions
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Has been cancelled
Deploy / build-worker (push) Has been cancelled
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 13:56:29 +00:00
1f66dfd400 feat(macro): ЕМИСС среднедушевые доходы (id=57039) -> macro_indicator (#946 part2)
All checks were successful
CI / changes (push) Successful in 6s
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
Deploy / changes (push) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
Deploy / build-backend (push) Successful in 1m31s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m33s
Deploy / deploy (push) Successful in 1m12s
CI / backend-tests (push) Successful in 6m23s
CI / backend-tests (pull_request) Successful in 6m31s
Расширяет Росстат-скрейпер ЕМИСС/fedstat-рядом «Среднедушевые денежные доходы
населения» (fedstat id=57039, OKATO 65=Свердл, квартальный, руб). Добавляет pure
SDMX-ML v1.0 GenericData-парсер (stdlib ElementTree, dev-тестируем на фикстурах) +
EMISS-ветку Celery-таски rosstat_macro_sync (open-data + ЕМИСС, per-source guard,
source='emiss', frequency параметризован, CAST not ::, SAVEPOINT per-row).
income_per_capita проброшен в site_finder/macro (region='sverdl').

ЕМИСС за WAF с dev -> fetch исполняется на проде (verified: httpx POST из
gendesign-worker-1 -> 200 + SDMX, парсер извлёк 9 Свердл-строк). ИПЦ (id=31074) /
индекс цен СМР НЕ landed: многомерные, дефолт-экспорт data.do без Свердл, нужна
dataGrid-выборка (selectedFilterIds + filter-tree AJAX + reCAPTCHA) -- задокументировано.

Refs #946.
2026-06-07 18:44:38 +05:00
f7c4d7a8c5 feat(macro): Росстат open-data macro scraper -> macro_indicator (#946)
All checks were successful
Deploy / changes (push) Successful in 7s
Deploy / build-backend (push) Successful in 1m50s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m54s
Deploy / deploy (push) Successful in 1m22s
EPIC2 macro-ingest: региональный (Свердловская обл. ОКТМО 65 / ЕКБ) скрейпер
Росстата в унифицированную macro_indicator (м.123). Зеркалит форму cbr_macro:
pure-парсеры + тонкий httpx, Celery-таска с SAVEPOINT-per-row upsert
(CAST not ::, ON CONFLICT по PK indicator_type/region/obs_date).

ЕМИСС/fedstat.ru (SDMX) за WAF (hard-403 на всех путях из dev-IP) -> ИПЦ/доходы/
СМР-цены пока недоступны (документировано в коде). rosstat.gov.ru/opendata
(стандарт 4.0) WAF-free -> приземляет демографию population_total
(sverdl 4.32M / ekb 1.41M, §7.11).

- services/scrapers/rosstat_emiss.py — fetch+parse open-data (registry->meta->data CSV)
- workers/tasks/rosstat_macro_sync.py — Celery upsert (source=rosstat)
- beat: rosstat-macro-sync-monthly; include в celery_app
- site_finder/macro.py: population_total -> region-aware default sverdl (additive)
- tests: 18 offline-тестов парсера + контракта таски

Без миграции (вписано в существующую схему), без новых зависимостей.

Refs #946.
2026-06-07 13:12:01 +00:00
50fcba1ca0 feat(sf): ingest ОКН-объектов ЕКБ из АИС ЕГРКН (точки+категория) → okn_objects (#1141)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m23s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m21s
Deploy / deploy (push) Successful in 1m15s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 12:51:18 +00:00
189ecc81f8 feat(sf): ingest функц.зон генплана ЕКБ-2045 из ГИСОГД-СО WFS → ekb_genplan_functional_zone (#1137)
All checks were successful
Deploy / build-backend (push) Successful in 1m18s
Deploy / build-frontend (push) Has been skipped
Deploy / changes (push) Successful in 5s
Deploy / build-worker (push) Successful in 2m28s
Deploy / deploy (push) Successful in 1m17s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 12:03:39 +00:00
1e53416277 feat(sf): ППТ/ПМТ ТЭП-парсер (Табл.11/12/13) → ekb_ppt_tep MVP (#1133)
All checks were successful
Deploy / changes (push) Successful in 7s
Deploy / build-backend (push) Successful in 1m29s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m27s
Deploy / deploy (push) Successful in 1m21s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 11:54:40 +00:00
57c6d119f5 feat(sf): scraper реестра КРТ-площадок ЕКБ → ekb_krt_sites (#1126)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 2m40s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 3m22s
Deploy / deploy (push) Successful in 1m30s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 11:24:49 +00:00
ef15190690 feat(sf): geocode geom для ekburg_construction_permits из МСК-66 (#1124)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m29s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m34s
Deploy / deploy (push) Successful in 1m21s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 11:16:45 +00:00
59f2628e0b feat(sf): ПАГЕ-парсер изъятия/резервирования → land_reservation (#1118)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 2m41s
Deploy / build-worker (push) Successful in 3m25s
Deploy / deploy (push) Successful in 1m25s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 10:15:03 +00:00
dcafb32f31 feat(sf): gknspecial_zone ИРД-harvest → ird_overlays (#1114)
Some checks failed
Deploy / deploy (push) Blocked by required conditions
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Has been cancelled
Deploy / build-backend (push) Has been cancelled
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 09:54:53 +00:00
e7a0af5ddc fix(sf): ekb_geoportal_client verify=False — росс-гос-CA не в trust-store контейнера (#1103)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m28s
Deploy / build-worker (push) Successful in 2m34s
Deploy / deploy (push) Successful in 1m11s
_http_get_json/_http_post_json → verify=False (const _VERIFY_TLS): геопортал ЕКБ отдаёт цепочку с росс-гос-CA, которого нет в trust-store контейнера → verify=True падал CERTIFICATE_VERIFY_FAILED, весь geoportal-слой (C8b/D9b/#1085) не работал бы с прода. Данные публичные open-data, секреты не передаются. Прецедент: nspd_lite/ekburg_permits/pzz_loader. Прод-probe подтвердил. +регрессия-тест.

Refs #1067.
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-06 20:10:26 +00:00
b1530467f7 feat(sf): opportunity-ЗУ harvest → ird_overlays (future-supply, #1100)
All checks were successful
Deploy / changes (push) Successful in 8s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m28s
Deploy / build-worker (push) Successful in 2m35s
Deploy / deploy (push) Successful in 1m10s
opportunity_harvest.py — grid-walk НСПД opportunity-слоёв (ЗУ аукцион/схема/свободные/межевание/ККР) по кварталам ЕКБ → ird_overlays с layer_kind='opportunity_*'. Переиспользует движок ird_harvest (#1090): _upsert_feature/_ekb_quarters. +krt_territories=37430 в LAYERS. Beat monthly (1-е 05:30). Self-contained, прод-прогон gated B6.

Closes #1086.
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-06 19:32:45 +00:00
b4e6e43713 feat(sf): EKB geoportal urbanCard — ПЗЗ-регламент content (#1094)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m30s
Deploy / build-worker (push) Successful in 2m55s
Deploy / deploy (push) Successful in 1m13s
Расширение ekb_geoportal_client.py — urbanCard pipeline для ПЗЗ-регламента (#1067 C8): searchByGeom → urbanCard → parse_zone_regulation → ZoneRegulation (zone_index + 3 списка ВРИ 540н + предельные параметры). Метод zone_regulation_at(lon,lat). Supplement к WFS (C7), аддитивно. 11 тестов. Wiring в C8b cache-таблицу.

Refs #1067, #1059.
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-06 19:05:47 +00:00
4e41f9be73 feat(sf): EKB geoportal WFS client (#1083)
Some checks failed
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m24s
Deploy / build-worker (push) Successful in 2m26s
Deploy / deploy (push) Has been cancelled
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
Новый ekb_geoportal_client.py — WFS-клиент муниципального геопортала ЕКБ (Geometa GeoServer, public). features_at_point (CQL INTERSECTS), features_in_bbox (CQL BBOX), zone_index_at (crosswalk координата→индекс ПЗЗ). EPSG:4326, OGC WFS GetFeature (не WMS — reproject-баг). Единый источник землепользование/КРТ/ППТ/ЗОУИТ для ЕКБ. 7 тестов. Foundation #1067 C7; wiring в C8/D9.

Refs #1067.
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-06 18:37:09 +00:00
c2e0428803 feat(nspd): get_feature_info FEATURE_COUNT=10 — multi-feature per point (#1080)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m14s
Deploy / build-worker (push) Successful in 2m24s
Deploy / deploy (push) Successful in 1m21s
get_feature_info() шлёт WMS FEATURE_COUNT=10 (новый параметр feature_count, default 10). NSPD по умолчанию отдаёт 1 фичу/точку → перекрывающиеся зоны одного слоя теряются (37581 «иные ЗОУИТ» на ЕКБ = 2 фичи). Tile-геометрия не тронута. +2 теста (multi-feature parse, override), существующий обновлён.

Refs #1067.
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-06 18:18:07 +00:00
38d948f078 fix(nspd): list_layers parse flat layers[] response, not children-tree (#1077)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m26s
Deploy / deploy (push) Successful in 1m9s
Deploy / build-worker (push) Successful in 2m27s
NSPDClient.list_layers() парсил _walk_layer_tree по id/children, но НСПД layers-theme-tree отдаёт слои в плоском layers[] с ключом layerId → list_layers() молча возвращал [] для любой темы. Fix: сначала плоский data["layers"], иначе fallback на _walk_layer_tree; defensive None-guard на missing id. +тест на flat-форму, backward-compat сохранён.

Refs #1067.
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-06 17:51:14 +00:00
420cda2831 feat(domrf): persist domrf_kn_objects.free_flats (#950 PR A) (#1002)
All checks were successful
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m26s
Deploy / build-worker (push) Successful in 2m27s
Deploy / deploy (push) Successful in 1m7s
Deploy / changes (push) Successful in 5s
2026-06-02 20:06:29 +00:00
c78c6ec2d0 feat(site-finder): live CBR key-rate scraper → macro_indicator (#945 PR B) (#964)
All checks were successful
Deploy / build-frontend (push) Has been skipped
Deploy / deploy (push) Successful in 1m10s
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m24s
Deploy / build-worker (push) Successful in 2m53s
2026-06-02 19:05:57 +00:00
4e4762f234 fix(tradein): backfill kn obj_class (parser + 3-pass) (#572) (#628)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m27s
Deploy / build-worker (push) Successful in 3m9s
Deploy / deploy (push) Successful in 1m9s
2026-05-28 15:04:32 +00:00
609ac8ea9d fix(tradein): backfill kn price_per_m2 via sale_graph + peer median (#573) (#625)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m43s
Deploy / build-worker (push) Successful in 2m41s
Deploy / deploy (push) Successful in 1m11s
2026-05-28 13:50:45 +00:00
099b8e4b59 fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503)
Some checks failed
Deploy / changes (push) Successful in 4s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m22s
Deploy / build-worker (push) Successful in 2m41s
Deploy / deploy (push) Failing after 35s
2026-05-24 10:53:28 +00:00
9ffeee7ed2 feat(22d): admin UI buttons (skip/force) + reduce catalog jitter 800→300ms (#352)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m48s
Deploy / build-worker (push) Successful in 3m1s
Deploy / build-frontend (push) Successful in 3m20s
Deploy / deploy (push) Successful in 55s
2026-05-17 23:38:22 +00:00
lekss361
e5498682ba fix(22d): commit outer transaction in scrape_catalog_objects orchestrator
PR #335 использовал `with db.begin_nested():` SAVEPOINT для per-row UPDATE,
но outer transaction никогда не commit'ился. После цикла Celery task
закрывал session (`db.close()` в finally) → autobegin'd outer tx
rollback → все UPDATE'ы теряются.

Симптом: task завершается с processed=3 succeeded=3, в Glitchtip нет ошибок,
но в БД ни одного catalog_scraped_at != NULL.

Fix: добавлен db.commit() после цикла + rollback на ошибку commit'а.
2026-05-18 01:10:06 +03:00
0567ad2130 feat(22d): domrf_catalog_object scraper — fill ~25 NULL kn_objects cols from SSR __NEXT_DATA__ (#335)
Some checks failed
Deploy / changes (push) Successful in 7s
Deploy / build-backend (push) Successful in 1m43s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m49s
Deploy / deploy (push) Failing after 44s
2026-05-17 21:26:22 +00:00
lekss361
2aa33e0eb8 fix(stealth): bump Semaphore 6->8 for higher concurrency in gather
Per-object 7 endpoints через asyncio.gather упираются в semaphore limit.
6 -> 8 — ещё +33% concurrency для batch fetch'а.
2026-05-17 23:04:51 +03:00
lekss361
0f289c05b1 fix(scrape-kn): real documents endpoints (5x) + skip obj_checks 404 + sem 3->6
PR #322 used placeholder URLs:
- /api/object/{id}/documents -> 404 (Playwright showed: 5 real endpoints:
  document/rpd / developer/report / project/documentation /
  documentation/other / document/permits). Replaced with 5 parallel
  fetches + merge into single upsert_documents call.
- /api/object/{id}/checks -> 404. 6 checkboxes likely inline in bulk
  kn/object payload; endpoint /checks does not exist. Commented out
  _fetch_obj_checks_safe with TODO for separate investigation PR.
- BrowserSession Semaphore(3) -> Semaphore(6) — double concurrency
  for intra-object gather.

GlitchTip: kn_scrape_failures run #19 showed 40 failures each for
both placeholder endpoints across first 30 objects of 1532.
2026-05-17 22:59:44 +03:00
f2983c86d7 feat(scrape-kn): parallel Phase B/C + wire obj_checks + documents + is_ekb auto-derive (#322)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 2m19s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m51s
Deploy / deploy (push) Successful in 55s
2026-05-17 19:40:16 +00:00
389e141ec9 fix(scrape-objective): stream-parse lots_pf via ijson to avoid OOM (#315)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 3m0s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 5m56s
Deploy / deploy (push) Successful in 1m19s
2026-05-17 16:37:30 +00:00
4bed58b43c feat(22d): domrf_catalog.py scraper for per-flat price/status from SSR HTML (#309)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m15s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m14s
Deploy / deploy (push) Successful in 1m0s
2026-05-17 15:40:01 +00:00
b37e011aaa feat(22c): domrf_kn_flat_plans new table + parser stub (#308)
Some checks failed
Deploy / build-frontend (push) Blocked by required conditions
Deploy / deploy (push) Blocked by required conditions
Deploy / changes (push) Successful in 5s
Deploy / build-worker (push) Has been cancelled
Deploy / build-backend (push) Has been cancelled
2026-05-17 15:39:34 +00:00
6612c1cd2b feat(22begh): schema extension — flats+6 / objects+30 / developers+4 / metro_top3 (#306)
All checks were successful
Deploy / build-backend (push) Successful in 1m32s
Deploy / build-frontend (push) Has been skipped
Deploy / changes (push) Successful in 5s
Deploy / build-worker (push) Successful in 3m32s
Deploy / deploy (push) Successful in 56s
2026-05-17 15:26:59 +00:00
171a029997 feat(22i): domrf_kn_documents new table + parser (#305)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m34s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m42s
Deploy / deploy (push) Successful in 59s
2026-05-17 15:21:31 +00:00
3183abe41b fix(22j): DEDUP domrf_kn_infrastructure + ON CONFLICT (issue #297) (#304)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m34s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m45s
Deploy / deploy (push) Successful in 1m4s
2026-05-17 15:15:03 +00:00
0b12a55eb6 fix(bug-22a): map missing kn-API fields (flat_number/living_area/wall_type/energy_eff) + backfill (#302)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m46s
Deploy / build-frontend (push) Has been skipped
Deploy / deploy (push) Successful in 58s
Deploy / build-worker (push) Successful in 2m50s
2026-05-17 14:59:02 +00:00
bfe16ae71b fix(sf-16): derive price_per_m2 from price_rub in _norm_flat + backfill SQL (#298)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m34s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m33s
Deploy / deploy (push) Successful in 58s
2026-05-17 14:32:23 +00:00
lekss361
3daf2a1fb1 fix(sf-15): extract obj_class from aiDescription (DOM.RF list API never returns objClass field)
Add _extract_obj_class_from_ai() with regex patterns for Комфорт/Бизнес/Премиум/Элит/Стандарт.
_norm_object() uses it as fallback; objClass from API still takes priority if ever returned.
Add backfill SQL 105 to retroactively populate existing rows from raw JSON payloads.
Coverage: ~1806/4548 rows will be filled; remainder NULL is expected (Нежилое/no AI desc).
2026-05-17 17:02:07 +03:00
lekss361
570c987e78 fix(#231): repair test_rate_limit_semaphore regression + docstring sync
Re-review feedback (PR #260 head 8191a85):

1. tests/scrapers/test_nspd_bulk_client.py:360 — test referenced bulk_mod._SEMAPHORE
   which was renamed to _SEMAPHORE_LIMIT in previous commit. AttributeError на
   полном pytest run. Заменено: assert bulk_mod._SEMAPHORE_LIMIT == 3 (smoke).
   max_concurrent <= 3 уже валидирует throttling — capacity check был лишним.

2. app/services/scrapers/nspd_client.py docstrings (lines 483, 506) — обновлены
   ссылки NSPDBulkClient._SEMAPHORE(3) → NSPDBulkClient._sem (per-instance,
   capacity=3) чтобы соответствовать новой архитектуре.
2026-05-17 10:22:16 +03:00
lekss361
fdced94626 fix(#231): switch search_by_quarter area layers to grid-walk dispatch (Sub-PR B)
- Add _GRID_WALK_LAYERS frozenset (territorial_zones, red_lines,
  engineering_structures, zouit_*, risk_*)
- _fetch_layer dispatch: area layers → get_features_in_bbox_grid(grid_n=7,
  step_m=50) [foundation из PR #247], EGRN layers (parcels, buildings) →
  legacy get_features_in_bbox
- Enrich engineering_structures features с classified_kind через
  classify_engineering_kind (PR #247 classifier)
- Per-layer logger.info heartbeat (method + count) для прогресса
- Bump harvest_quarter soft_time_limit 120 → 600s
  (grid-walk: 11 layers × 49 cells × ~70ms ≈ 40s + retries margin)
- TestFetchLayerDispatch (3 tests): 34 passed, 1 skipped

Pre-push deep-code-reviewer: APPROVE (1 medium nit — opportunity_oopt не в
grid-walk, защищено через include_opportunity=False default; follow-up issue
рекомендован).
2026-05-17 09:46:42 +03:00
lekss361
4a48d49474 fix(#242): ekburg_permits SSL — verify=False для CA Минцифры РФ 2026-05-17 08:57:42 +03:00
4519daf149 feat(#126): Sub-PR A — NSPDClient.get_features_in_bbox_grid + engineering classifier (#247)
All checks were successful
Deploy / build-backend (push) Successful in 1m23s
Deploy / build-frontend (push) Has been skipped
Deploy / deploy (push) Successful in 48s
Deploy / changes (push) Successful in 5s
Deploy / build-worker (push) Successful in 2m18s
2026-05-17 05:54:30 +00:00
lekss361
db1ff5b1fc feat(nspd): denorm nspd_parcels + nspd_buildings tables (#94 PR3 of 4)
- data/sql/99_nspd_entities_denorm.sql: 2 tables (cad_num PK) + 6 indexes
  (quarter_cad, GIST geom, partial WHERE purpose ILIKE '%многокв%')
- nspd_denorm.py: denorm_parcel_feature / denorm_building_feature / denorm_dump
  - SAVEPOINT per row via with db.begin_nested()
  - ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON(:geom), 3857), 4326)
  - json.dumps(geom, ensure_ascii=False) for psycopg bind
  - All CAST(:x AS type) — no :type
- nspd_sync.py: inline denorm after _upsert_dump in harvest_quarter (non-fatal)
- nspd_denorm_backfill.py: Celery one-shot task (per-quarter commit)
- admin_etl.py: POST /api/v1/admin/etl/nspd-denorm-backfill (AdminTokenAuth)
- 19 tests: coerce helpers, denorm parcel/building, aggregate counts

Foundation for downstream features (МКД neighbors lookup, parcel attrs).

Part of #94
2026-05-16 20:20:03 +03:00
lekss361
ed3c128528 feat(nspd): TIER 4 opportunity layers + red lines (#94 PR2 of 4)
- NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt)
  + QuarterDump.opportunity field
- nspd_sync: harvest_quarter accepts include_opportunity, denorm cols
  has_auction_parcels + opportunity_count in UPSERT
- quarter_dump_lookup:
  - _get_opportunity_parcels (sort by distance, early-exit on count=0)
  - _get_red_lines (query existing dump.red_lines core path, layer='red_lines')
- SQL 89_*: has_auction_parcels + opportunity_count + partial index
- Pydantic: OpportunityParcel + RedLine schemas
- /analyze: nspd_opportunity_parcels + nspd_red_lines fields
- Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration
- Tests: 28 total (11 PR1 + 17 PR2), all pass

Red lines uses existing core harvest path (layer 879243 already in dump.red_lines
from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed.

Part of #94
2026-05-16 19:06:22 +03:00
adc45f5724 feat(permits): ЕКБ РНС/РВЭ xlsx parser + schema + ETL (#105 PR A) (#211)
All checks were successful
Deploy / changes (push) Successful in 4s
Deploy / build-backend (push) Successful in 1m29s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m42s
Deploy / deploy (push) Successful in 39s
2026-05-16 11:41:21 +00:00