a49ee2ccfc
fix(domrf_kn): детерминированный sha256-fallback для flat_id ( #1208 )
...
CI / changes (push) Successful in 6s
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (push) Successful in 6m27s
CI / backend-tests (pull_request) Successful in 6m27s
Fallback flat_id в _norm_flat использовал abs(hash(elem)) % (2**63 - 1).
В CPython hash(str) РАНДОМИЗИРОВАН per-process (PYTHONHASHSEED нигде в
репо не зафиксирован — uvicorn/celery не выставляют его). Эффект:
- При resume упавшего sweep (resume_kn_run в новом процессе воркера) до
10 объектов после checkpoint перечитываются. Квартиры без flatId
получают ДРУГОЙ hash-id → ON CONFLICT (id, snapshot_date) не
срабатывает → дубли строк одной квартиры в одном snapshot.
- То же при повторном прогоне за ту же дату — каждый раз новый id.
- Дубли инфлируют все агрегации (units_sold/price медианы, supply_layers).
Patch:
- abs(hash(elem)) → int.from_bytes(sha256(elem)[:8], "big") % (2**63 - 1).
- sha256 стабилен между процессами/перезапусками. 8 байт → BIGINT-fit.
6 новых юнит-тестов (test_domrf_kn_normalize.py): formula matches sha256,
stable across calls, distinct elems→distinct ids, flatId wins over
fallback, no-id→None, BIGINT-fit. ruff clean.
Closes #1208
2026-06-13 11:01:41 +05:00
f9e045028c
feat(macro): land construction price index (СМР) via open Rosstat xlsx ( #946 )
...
CI / changes (push) Successful in 6s
CI / changes (pull_request) Successful in 7s
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (pull_request) Successful in 6m27s
Deploy / build-frontend (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / backend-tests (push) Successful in 6m29s
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m43s
Deploy / build-worker (push) Successful in 3m3s
Deploy / deploy (push) Successful in 1m15s
Closes the last #946 gap (СМР). Fetches «Индексы цен производителей на
строительную продукцию по РФ» from the OPEN Rosstat xlsx
(rosstat.gov.ru/storage/mediabank/Invest_ind_stroitel_MM-YYYY.xlsx),
parses the monthly «к предыдущему месяцу» section → macro_indicator
(indicator_type='construction_price_index', region='rf', monthly, %).
The reCAPTCHA-gated fedstat dataGrid.do path is DELIBERATELY NOT used —
the open Rosstat source needs no captcha. EMISS id=31108 documented as a
prod-only SDMX alternative (not wired).
URL is date-derived with bounded look-back (filename changes monthly;
_fetch_construction_latest walks current month back to 6, first 200 wins;
year-boundary handled, exhaustion raises, 404 breaks fast — no hang).
Wired into rosstat_macro_sync with per-source try/except (can't break
demography/income) + SAVEPOINT-per-row. None-not-0 on blank months.
33 tests (incl look-back + year-boundary + graceful), ruff clean.
Note: no construction-cost channel in macro_coefficient yet (demand-side
only) — series is ingested/available for future cost-side use + display.
Refs #946
2026-06-09 11:28:05 +05:00
25e21c2bff
feat(macro): CBR inflation (ИПЦ YoY) -> macro_indicator + activate §9.5 channel ( #946 )
...
CI / frontend-tests (pull_request) Has been skipped
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m45s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m42s
CI / changes (push) Successful in 6s
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
Deploy / deploy (push) Successful in 1m15s
CI / backend-tests (push) Successful in 6m30s
CI / backend-tests (pull_request) Successful in 6m32s
fedstat ИПЦ is reCAPTCHA-blocked; CBR publishes inflation openly. Add
fetch_inflation + parse_inflation_xlsx (CBR UniDbQuery DownloadExcel/132934,
monthly % г/г, region=rf, source=cbr) to cbr_macro.py; upsert
indicator_type=inflation_yoy via the existing cbr_macro_sync task (per-series
guard, SAVEPOINT-per-row, CAST not ::, ON CONFLICT on the PK).
Surface inflation_yoy in MonthlyMacro (frozen, carry-forward) and ACTIVATE the
reserved §9.5 inflation channel (macro_coefficient f_inflation: level-vs-4%-target
nudge, non-positive to avoid double-counting f_rate, excluded from
_RATE_DRIVEN_FACTORS). Channel was DEGRADED (no data) -> now BACKED + consumed;
_CONF_HIGH_MIN_BACKED 4->5. Deterministic (§16/§26); renorm claims the reserved
0.08 slice as designed. Live-verified (2026-04 5.58%); 194 macro + 902 forecasting
tests green. No migration, no new deps.
Refs #946 .
2026-06-08 11:41:14 +05:00
5da4271235
feat(sf): OCR-пайплайн изъятия ЕКБ (Tesseract rus) → land_reservation ( #1062 )
...
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / changes (push) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (push) Successful in 6m22s
CI / backend-tests (pull_request) Successful in 6m19s
- Dockerfile: добавлены tesseract-ocr tesseract-ocr-rus в runner apt-слой
- pyproject.toml: pymupdf>=1.24 (рендер PDF без poppler) + pytesseract>=0.3.13
- izyatie_client.py: list_izyatie_documents() + fetch_pdf() с екатеринбург.рф
- izyatie_ocr.py: ocr_pdf_text() (PyMuPDF→PIL→Tesseract rus) + extract_izyatie_records()
с нормализацией OCR-шума (пробелы в кад-номерах, кирилл. О→0, б→6)
- izyatie_ocr_ingest.py: Celery task → land_reservation UPSERT (SAVEPOINT per-row)
- beat_schedule.py: izyatie-ocr-ingest-weekly (пятница 07:00 МСК)
- 33 теста, все зелёные; без реального Tesseract/сети в CI
2026-06-07 18:07:06 +03:00
e2faab801c
fix(sf): pravo.gov66 law_type числовой код =1 Постановление ( #1149 )
...
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m30s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m21s
Deploy / deploy (push) Successful in 1m15s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-06-07 14:17:52 +00:00
a6e100b52a
fix(sf): корректное urlencode form-data в okn_egrkn_client ( #1147 )
...
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m29s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m39s
Deploy / deploy (push) Successful in 1m14s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-06-07 14:13:34 +00:00
9ee35bebf0
fix(sf): User-Agent header for okn-mk searchMap (bypass 403) ( #1146 )
...
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m14s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m30s
Deploy / deploy (push) Successful in 1m10s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-06-07 13:57:34 +00:00
396db20c48
feat(sf): enumerate изъятия/резервирования с pravo.gov66 → land_reservation ( #1145 )
...
Deploy / build-frontend (push) Blocked by required conditions
Deploy / deploy (push) Blocked by required conditions
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Has been cancelled
Deploy / build-worker (push) Has been cancelled
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 13:56:29 +00:00
1f66dfd400
feat(macro): ЕМИСС среднедушевые доходы (id=57039) -> macro_indicator ( #946 part2)
...
CI / changes (push) Successful in 6s
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
Deploy / changes (push) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
Deploy / build-backend (push) Successful in 1m31s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m33s
Deploy / deploy (push) Successful in 1m12s
CI / backend-tests (push) Successful in 6m23s
CI / backend-tests (pull_request) Successful in 6m31s
Расширяет Росстат-скрейпер ЕМИСС/fedstat-рядом «Среднедушевые денежные доходы
населения» (fedstat id=57039, OKATO 65=Свердл, квартальный, руб). Добавляет pure
SDMX-ML v1.0 GenericData-парсер (stdlib ElementTree, dev-тестируем на фикстурах) +
EMISS-ветку Celery-таски rosstat_macro_sync (open-data + ЕМИСС, per-source guard,
source='emiss', frequency параметризован, CAST not ::, SAVEPOINT per-row).
income_per_capita проброшен в site_finder/macro (region='sverdl').
ЕМИСС за WAF с dev -> fetch исполняется на проде (verified: httpx POST из
gendesign-worker-1 -> 200 + SDMX, парсер извлёк 9 Свердл-строк). ИПЦ (id=31074) /
индекс цен СМР НЕ landed: многомерные, дефолт-экспорт data.do без Свердл, нужна
dataGrid-выборка (selectedFilterIds + filter-tree AJAX + reCAPTCHA) -- задокументировано.
Refs #946 .
2026-06-07 18:44:38 +05:00
f7c4d7a8c5
feat(macro): Росстат open-data macro scraper -> macro_indicator ( #946 )
...
Deploy / changes (push) Successful in 7s
Deploy / build-backend (push) Successful in 1m50s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m54s
Deploy / deploy (push) Successful in 1m22s
EPIC2 macro-ingest: региональный (Свердловская обл. ОКТМО 65 / ЕКБ) скрейпер
Росстата в унифицированную macro_indicator (м.123). Зеркалит форму cbr_macro:
pure-парсеры + тонкий httpx, Celery-таска с SAVEPOINT-per-row upsert
(CAST not ::, ON CONFLICT по PK indicator_type/region/obs_date).
ЕМИСС/fedstat.ru (SDMX) за WAF (hard-403 на всех путях из dev-IP) -> ИПЦ/доходы/
СМР-цены пока недоступны (документировано в коде). rosstat.gov.ru/opendata
(стандарт 4.0) WAF-free -> приземляет демографию population_total
(sverdl 4.32M / ekb 1.41M, §7.11).
- services/scrapers/rosstat_emiss.py — fetch+parse open-data (registry->meta->data CSV)
- workers/tasks/rosstat_macro_sync.py — Celery upsert (source=rosstat)
- beat: rosstat-macro-sync-monthly; include в celery_app
- site_finder/macro.py: population_total -> region-aware default sverdl (additive)
- tests: 18 offline-тестов парсера + контракта таски
Без миграции (вписано в существующую схему), без новых зависимостей.
Refs #946 .
2026-06-07 13:12:01 +00:00
50fcba1ca0
feat(sf): ingest ОКН-объектов ЕКБ из АИС ЕГРКН (точки+категория) → okn_objects ( #1141 )
...
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m23s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m21s
Deploy / deploy (push) Successful in 1m15s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 12:51:18 +00:00
189ecc81f8
feat(sf): ingest функц.зон генплана ЕКБ-2045 из ГИСОГД-СО WFS → ekb_genplan_functional_zone ( #1137 )
...
Deploy / build-backend (push) Successful in 1m18s
Deploy / build-frontend (push) Has been skipped
Deploy / changes (push) Successful in 5s
Deploy / build-worker (push) Successful in 2m28s
Deploy / deploy (push) Successful in 1m17s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 12:03:39 +00:00
1e53416277
feat(sf): ППТ/ПМТ ТЭП-парсер (Табл.11/12/13) → ekb_ppt_tep MVP ( #1133 )
...
Deploy / changes (push) Successful in 7s
Deploy / build-backend (push) Successful in 1m29s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m27s
Deploy / deploy (push) Successful in 1m21s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 11:54:40 +00:00
57c6d119f5
feat(sf): scraper реестра КРТ-площадок ЕКБ → ekb_krt_sites ( #1126 )
...
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 2m40s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 3m22s
Deploy / deploy (push) Successful in 1m30s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 11:24:49 +00:00
ef15190690
feat(sf): geocode geom для ekburg_construction_permits из МСК-66 ( #1124 )
...
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m29s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m34s
Deploy / deploy (push) Successful in 1m21s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 11:16:45 +00:00
59f2628e0b
feat(sf): ПАГЕ-парсер изъятия/резервирования → land_reservation ( #1118 )
...
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 2m41s
Deploy / build-worker (push) Successful in 3m25s
Deploy / deploy (push) Successful in 1m25s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 10:15:03 +00:00
dcafb32f31
feat(sf): gknspecial_zone ИРД-harvest → ird_overlays ( #1114 )
...
Deploy / deploy (push) Blocked by required conditions
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Has been cancelled
Deploy / build-backend (push) Has been cancelled
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-07 09:54:53 +00:00
e7a0af5ddc
fix(sf): ekb_geoportal_client verify=False — росс-гос-CA не в trust-store контейнера ( #1103 )
...
Deploy / changes (push) Successful in 5s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m28s
Deploy / build-worker (push) Successful in 2m34s
Deploy / deploy (push) Successful in 1m11s
_http_get_json/_http_post_json → verify=False (const _VERIFY_TLS): геопортал ЕКБ отдаёт цепочку с росс-гос-CA, которого нет в trust-store контейнера → verify=True падал CERTIFICATE_VERIFY_FAILED, весь geoportal-слой (C8b/D9b/#1085) не работал бы с прода. Данные публичные open-data, секреты не передаются. Прецедент: nspd_lite/ekburg_permits/pzz_loader. Прод-probe подтвердил. +регрессия-тест.
Refs #1067 .
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-06 20:10:26 +00:00
b1530467f7
feat(sf): opportunity-ЗУ harvest → ird_overlays (future-supply, #1100 )
...
Deploy / changes (push) Successful in 8s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m28s
Deploy / build-worker (push) Successful in 2m35s
Deploy / deploy (push) Successful in 1m10s
opportunity_harvest.py — grid-walk НСПД opportunity-слоёв (ЗУ аукцион/схема/свободные/межевание/ККР) по кварталам ЕКБ → ird_overlays с layer_kind='opportunity_*'. Переиспользует движок ird_harvest (#1090 ): _upsert_feature/_ekb_quarters. +krt_territories=37430 в LAYERS. Beat monthly (1-е 05:30). Self-contained, прод-прогон gated B6.
Closes #1086 .
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-06 19:32:45 +00:00
b4e6e43713
feat(sf): EKB geoportal urbanCard — ПЗЗ-регламент content ( #1094 )
...
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m30s
Deploy / build-worker (push) Successful in 2m55s
Deploy / deploy (push) Successful in 1m13s
Расширение ekb_geoportal_client.py — urbanCard pipeline для ПЗЗ-регламента (#1067 C8): searchByGeom → urbanCard → parse_zone_regulation → ZoneRegulation (zone_index + 3 списка ВРИ 540н + предельные параметры). Метод zone_regulation_at(lon,lat). Supplement к WFS (C7), аддитивно. 11 тестов. Wiring в C8b cache-таблицу.
Refs #1067 , #1059 .
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-06 19:05:47 +00:00
4e41f9be73
feat(sf): EKB geoportal WFS client ( #1083 )
...
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m24s
Deploy / build-worker (push) Successful in 2m26s
Deploy / deploy (push) Has been cancelled
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
Новый ekb_geoportal_client.py — WFS-клиент муниципального геопортала ЕКБ (Geometa GeoServer, public). features_at_point (CQL INTERSECTS), features_in_bbox (CQL BBOX), zone_index_at (crosswalk координата→индекс ПЗЗ). EPSG:4326, OGC WFS GetFeature (не WMS — reproject-баг). Единый источник землепользование/КРТ/ППТ/ЗОУИТ для ЕКБ. 7 тестов. Foundation #1067 C7; wiring в C8/D9.
Refs #1067 .
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-06 18:37:09 +00:00
c2e0428803
feat(nspd): get_feature_info FEATURE_COUNT=10 — multi-feature per point ( #1080 )
...
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m14s
Deploy / build-worker (push) Successful in 2m24s
Deploy / deploy (push) Successful in 1m21s
get_feature_info() шлёт WMS FEATURE_COUNT=10 (новый параметр feature_count, default 10). NSPD по умолчанию отдаёт 1 фичу/точку → перекрывающиеся зоны одного слоя теряются (37581 «иные ЗОУИТ» на ЕКБ = 2 фичи). Tile-геометрия не тронута. +2 теста (multi-feature parse, override), существующий обновлён.
Refs #1067 .
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-06 18:18:07 +00:00
38d948f078
fix(nspd): list_layers parse flat layers[] response, not children-tree ( #1077 )
...
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m26s
Deploy / deploy (push) Successful in 1m9s
Deploy / build-worker (push) Successful in 2m27s
NSPDClient.list_layers() парсил _walk_layer_tree по id/children, но НСПД layers-theme-tree отдаёт слои в плоском layers[] с ключом layerId → list_layers() молча возвращал [] для любой темы. Fix: сначала плоский data["layers"], иначе fallback на _walk_layer_tree; defensive None-guard на missing id. +тест на flat-форму, backward-compat сохранён.
Refs #1067 .
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-06 17:51:14 +00:00
420cda2831
feat(domrf): persist domrf_kn_objects.free_flats ( #950 PR A) ( #1002 )
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m26s
Deploy / build-worker (push) Successful in 2m27s
Deploy / deploy (push) Successful in 1m7s
Deploy / changes (push) Successful in 5s
2026-06-02 20:06:29 +00:00
c78c6ec2d0
feat(site-finder): live CBR key-rate scraper → macro_indicator ( #945 PR B) ( #964 )
Deploy / build-frontend (push) Has been skipped
Deploy / deploy (push) Successful in 1m10s
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m24s
Deploy / build-worker (push) Successful in 2m53s
2026-06-02 19:05:57 +00:00
4e4762f234
fix(tradein): backfill kn obj_class (parser + 3-pass) ( #572 ) ( #628 )
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m27s
Deploy / build-worker (push) Successful in 3m9s
Deploy / deploy (push) Successful in 1m9s
2026-05-28 15:04:32 +00:00
609ac8ea9d
fix(tradein): backfill kn price_per_m2 via sale_graph + peer median ( #573 ) ( #625 )
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m43s
Deploy / build-worker (push) Successful in 2m41s
Deploy / deploy (push) Successful in 1m11s
2026-05-28 13:50:45 +00:00
099b8e4b59
fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints ( #503 )
Deploy / changes (push) Successful in 4s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m22s
Deploy / build-worker (push) Successful in 2m41s
Deploy / deploy (push) Failing after 35s
2026-05-24 10:53:28 +00:00
9ffeee7ed2
feat(22d): admin UI buttons (skip/force) + reduce catalog jitter 800→300ms ( #352 )
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m48s
Deploy / build-worker (push) Successful in 3m1s
Deploy / build-frontend (push) Successful in 3m20s
Deploy / deploy (push) Successful in 55s
2026-05-17 23:38:22 +00:00
lekss361
e5498682ba
fix(22d): commit outer transaction in scrape_catalog_objects orchestrator
...
PR #335 использовал `with db.begin_nested():` SAVEPOINT для per-row UPDATE,
но outer transaction никогда не commit'ился. После цикла Celery task
закрывал session (`db.close()` в finally) → autobegin'd outer tx
rollback → все UPDATE'ы теряются.
Симптом: task завершается с processed=3 succeeded=3, в Glitchtip нет ошибок,
но в БД ни одного catalog_scraped_at != NULL.
Fix: добавлен db.commit() после цикла + rollback на ошибку commit'а.
2026-05-18 01:10:06 +03:00
0567ad2130
feat(22d): domrf_catalog_object scraper — fill ~25 NULL kn_objects cols from SSR __NEXT_DATA__ ( #335 )
Deploy / changes (push) Successful in 7s
Deploy / build-backend (push) Successful in 1m43s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m49s
Deploy / deploy (push) Failing after 44s
2026-05-17 21:26:22 +00:00
lekss361
2aa33e0eb8
fix(stealth): bump Semaphore 6->8 for higher concurrency in gather
...
Per-object 7 endpoints через asyncio.gather упираются в semaphore limit.
6 -> 8 — ещё +33% concurrency для batch fetch'а.
2026-05-17 23:04:51 +03:00
lekss361
0f289c05b1
fix(scrape-kn): real documents endpoints (5x) + skip obj_checks 404 + sem 3->6
...
PR #322 used placeholder URLs:
- /api/object/{id}/documents -> 404 (Playwright showed: 5 real endpoints:
document/rpd / developer/report / project/documentation /
documentation/other / document/permits). Replaced with 5 parallel
fetches + merge into single upsert_documents call.
- /api/object/{id}/checks -> 404. 6 checkboxes likely inline in bulk
kn/object payload; endpoint /checks does not exist. Commented out
_fetch_obj_checks_safe with TODO for separate investigation PR.
- BrowserSession Semaphore(3) -> Semaphore(6) — double concurrency
for intra-object gather.
GlitchTip: kn_scrape_failures run #19 showed 40 failures each for
both placeholder endpoints across first 30 objects of 1532.
2026-05-17 22:59:44 +03:00
f2983c86d7
feat(scrape-kn): parallel Phase B/C + wire obj_checks + documents + is_ekb auto-derive ( #322 )
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 2m19s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m51s
Deploy / deploy (push) Successful in 55s
2026-05-17 19:40:16 +00:00
389e141ec9
fix(scrape-objective): stream-parse lots_pf via ijson to avoid OOM ( #315 )
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 3m0s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 5m56s
Deploy / deploy (push) Successful in 1m19s
2026-05-17 16:37:30 +00:00
4bed58b43c
feat(22d): domrf_catalog.py scraper for per-flat price/status from SSR HTML ( #309 )
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m15s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m14s
Deploy / deploy (push) Successful in 1m0s
2026-05-17 15:40:01 +00:00
b37e011aaa
feat(22c): domrf_kn_flat_plans new table + parser stub ( #308 )
Deploy / build-frontend (push) Blocked by required conditions
Deploy / deploy (push) Blocked by required conditions
Deploy / changes (push) Successful in 5s
Deploy / build-worker (push) Has been cancelled
Deploy / build-backend (push) Has been cancelled
2026-05-17 15:39:34 +00:00
6612c1cd2b
feat(22begh): schema extension — flats+6 / objects+30 / developers+4 / metro_top3 ( #306 )
Deploy / build-backend (push) Successful in 1m32s
Deploy / build-frontend (push) Has been skipped
Deploy / changes (push) Successful in 5s
Deploy / build-worker (push) Successful in 3m32s
Deploy / deploy (push) Successful in 56s
2026-05-17 15:26:59 +00:00
171a029997
feat(22i): domrf_kn_documents new table + parser ( #305 )
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m34s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m42s
Deploy / deploy (push) Successful in 59s
2026-05-17 15:21:31 +00:00
3183abe41b
fix(22j): DEDUP domrf_kn_infrastructure + ON CONFLICT (issue #297 ) ( #304 )
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m34s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m45s
Deploy / deploy (push) Successful in 1m4s
2026-05-17 15:15:03 +00:00
0b12a55eb6
fix(bug-22a): map missing kn-API fields (flat_number/living_area/wall_type/energy_eff) + backfill ( #302 )
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m46s
Deploy / build-frontend (push) Has been skipped
Deploy / deploy (push) Successful in 58s
Deploy / build-worker (push) Successful in 2m50s
2026-05-17 14:59:02 +00:00
bfe16ae71b
fix(sf-16): derive price_per_m2 from price_rub in _norm_flat + backfill SQL ( #298 )
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m34s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m33s
Deploy / deploy (push) Successful in 58s
2026-05-17 14:32:23 +00:00
lekss361
3daf2a1fb1
fix(sf-15): extract obj_class from aiDescription (DOM.RF list API never returns objClass field)
...
Add _extract_obj_class_from_ai() with regex patterns for Комфорт/Бизнес/Премиум/Элит/Стандарт.
_norm_object() uses it as fallback; objClass from API still takes priority if ever returned.
Add backfill SQL 105 to retroactively populate existing rows from raw JSON payloads.
Coverage: ~1806/4548 rows will be filled; remainder NULL is expected (Нежилое/no AI desc).
2026-05-17 17:02:07 +03:00
lekss361
570c987e78
fix( #231 ): repair test_rate_limit_semaphore regression + docstring sync
...
Re-review feedback (PR #260 head 8191a85 ):
1. tests/scrapers/test_nspd_bulk_client.py:360 — test referenced bulk_mod._SEMAPHORE
which was renamed to _SEMAPHORE_LIMIT in previous commit. AttributeError на
полном pytest run. Заменено: assert bulk_mod._SEMAPHORE_LIMIT == 3 (smoke).
max_concurrent <= 3 уже валидирует throttling — capacity check был лишним.
2. app/services/scrapers/nspd_client.py docstrings (lines 483, 506) — обновлены
ссылки NSPDBulkClient._SEMAPHORE(3) → NSPDBulkClient._sem (per-instance,
capacity=3) чтобы соответствовать новой архитектуре.
2026-05-17 10:22:16 +03:00
lekss361
fdced94626
fix( #231 ): switch search_by_quarter area layers to grid-walk dispatch (Sub-PR B)
...
- Add _GRID_WALK_LAYERS frozenset (territorial_zones, red_lines,
engineering_structures, zouit_*, risk_*)
- _fetch_layer dispatch: area layers → get_features_in_bbox_grid(grid_n=7,
step_m=50) [foundation из PR #247 ], EGRN layers (parcels, buildings) →
legacy get_features_in_bbox
- Enrich engineering_structures features с classified_kind через
classify_engineering_kind (PR #247 classifier)
- Per-layer logger.info heartbeat (method + count) для прогресса
- Bump harvest_quarter soft_time_limit 120 → 600s
(grid-walk: 11 layers × 49 cells × ~70ms ≈ 40s + retries margin)
- TestFetchLayerDispatch (3 tests): 34 passed, 1 skipped
Pre-push deep-code-reviewer: APPROVE (1 medium nit — opportunity_oopt не в
grid-walk, защищено через include_opportunity=False default; follow-up issue
рекомендован).
2026-05-17 09:46:42 +03:00
lekss361
4a48d49474
fix( #242 ): ekburg_permits SSL — verify=False для CA Минцифры РФ
2026-05-17 08:57:42 +03:00
4519daf149
feat( #126 ): Sub-PR A — NSPDClient.get_features_in_bbox_grid + engineering classifier ( #247 )
Deploy / build-backend (push) Successful in 1m23s
Deploy / build-frontend (push) Has been skipped
Deploy / deploy (push) Successful in 48s
Deploy / changes (push) Successful in 5s
Deploy / build-worker (push) Successful in 2m18s
2026-05-17 05:54:30 +00:00
lekss361
db1ff5b1fc
feat(nspd): denorm nspd_parcels + nspd_buildings tables ( #94 PR3 of 4)
...
- data/sql/99_nspd_entities_denorm.sql: 2 tables (cad_num PK) + 6 indexes
(quarter_cad, GIST geom, partial WHERE purpose ILIKE '%многокв%')
- nspd_denorm.py: denorm_parcel_feature / denorm_building_feature / denorm_dump
- SAVEPOINT per row via with db.begin_nested()
- ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON(:geom), 3857), 4326)
- json.dumps(geom, ensure_ascii=False) for psycopg bind
- All CAST(:x AS type) — no ❌ :type
- nspd_sync.py: inline denorm after _upsert_dump in harvest_quarter (non-fatal)
- nspd_denorm_backfill.py: Celery one-shot task (per-quarter commit)
- admin_etl.py: POST /api/v1/admin/etl/nspd-denorm-backfill (AdminTokenAuth)
- 19 tests: coerce helpers, denorm parcel/building, aggregate counts
Foundation for downstream features (МКД neighbors lookup, parcel attrs).
Part of #94
2026-05-16 20:20:03 +03:00
lekss361
ed3c128528
feat(nspd): TIER 4 opportunity layers + red lines ( #94 PR2 of 4)
...
- NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt)
+ QuarterDump.opportunity field
- nspd_sync: harvest_quarter accepts include_opportunity, denorm cols
has_auction_parcels + opportunity_count in UPSERT
- quarter_dump_lookup:
- _get_opportunity_parcels (sort by distance, early-exit on count=0)
- _get_red_lines (query existing dump.red_lines core path, layer='red_lines')
- SQL 89_*: has_auction_parcels + opportunity_count + partial index
- Pydantic: OpportunityParcel + RedLine schemas
- /analyze: nspd_opportunity_parcels + nspd_red_lines fields
- Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration
- Tests: 28 total (11 PR1 + 17 PR2), all pass
Red lines uses existing core harvest path (layer 879243 already in dump.red_lines
from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed.
Part of #94
2026-05-16 19:06:22 +03:00
adc45f5724
feat(permits): ЕКБ РНС/РВЭ xlsx parser + schema + ETL ( #105 PR A) ( #211 )
Deploy / changes (push) Successful in 4s
Deploy / build-backend (push) Successful in 1m29s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m42s
Deploy / deploy (push) Successful in 39s
2026-05-16 11:41:21 +00:00