88 commits

Author	SHA1	Message	Date
bot-backend	3421b21632	fix(scrapers): разблокировать domrf_kn flats — изоляция flats от poison-extras + throttle (#1945) (#2050)	2026-06-28 13:03:43 +00:00
bot-backend	f091554942	fix(scrapers): named BrowserSession concurrency const, fix Semaphore(3) doc drift (#1347) (#1913) ... named BrowserSession concurrency const + fix Semaphore(3) doc drift (#1347): keeps 8 (issue-sanctioned doc option). Refs #1347	2026-06-26 07:44:29 +00:00
bot-backend	0a72ef9491	merge main + renumber 163→167 + guard riasurt harvest (#108 review)	2026-06-17 22:52:15 +03:00
bot-backend	31e316e04e	feat(site-finder): РИАСУРТ Свердл 14-layer harvest client+schema+task for ЕКБ agglomeration (#108)	2026-06-17 21:49:04 +03:00
lekss361	0fb70f2f90	Merge pull request 'fix(sql): EMISS PK includes period_type so yearly+Q1 coexist (#1606 follow-up)' (#1706) from fix/emiss-pk-granularity-1687 into main	2026-06-17 18:46:00 +00:00
bot-backend	5170107405	fix(sql): EMISS PK includes period_type so yearly+Q1 coexist (#1606 follow-up) ... Adds period_type TEXT NOT NULL DEFAULT 'unknown' to macro_indicator and widens the PRIMARY KEY to (indicator_type, region, obs_date, period_type). Before: yearly aggregate ('год'→obs_date YYYY-01-01) and Q1 ('I квартал'→ same date) shared the same PK slot → ON CONFLICT DO UPDATE made them overwrite each other despite the in-memory dedup fix in #1687. After: each EmissRow carries period_type from _emiss_period_granularity ('year'/'quarter'/'month'); DB ON CONFLICT targets include it, so yearly+Q1 rows genuinely coexist. Non-EMISS sources (CBR, rosstat open-data, domrf) use period_type='unknown' (literal in SQL) — they are disambiguated by obs_date already, so the wider PK is backward-compatible. All ON CONFLICT clauses in cbr_macro_sync.py and rosstat_macro_sync.py updated accordingly. Migration: 163_emiss_pk_period_type.sql (idempotent BEGIN/COMMIT). Tests: 49 passed (emiss + rosstat suite), ruff clean, py_compile OK.	2026-06-17 21:27:34 +03:00
bot-backend	c3c89646ce	fix(scrapers): narrow domrf Level-1 status badge-class regex (#1609 follow-up) ... _STATUS_BADGE_CLS_RE was too broad: `status|badge|tag|chip|label` matched generic UI elements (e.g. <span class="label">, <span class="chip">) that are unrelated to the sale-status badge, risking picking the wrong block. Narrowed to `(?<![a-z])status(?![a-z])` — requires the literal token "status" as a hyphen-delimited component of the CSS class (matches `status-badge`, `flat-status-tag`, `object-status` but not bare `label`, `tag`, `chip`, `badge`). Adds regression test: page with generic label/tag/chip/badge elements containing "В продаже" must NOT activate Level-1; only the real `status-badge` block ("Продана") should be returned → status=sold.	2026-06-17 21:21:56 +03:00
lekss361	9a0ca9336f	Merge pull request 'fix(scrapers): scope+stem domrf status classification (#1609)' (#1686) from fix/domrf-status-classify-1609 into main	2026-06-17 17:55:42 +00:00
bot-backend	4f0cff92ec	fix(scrapers): scope+stem domrf status classification, fix продано/продана mismatch (#1609) ... Replace whole-HTML re.search for status with a 3-level block-scoped strategy: 1. CSS badge classes (status/badge/tag/chip/label) — highest precision. 2. Proximity to block labelled «Статус» via _find_text_near. 3. Full blocks scan where sold/reserved keywords always beat free — preventing «в продаже» from nav/similar-flats sections from misclassifying sold flats as free. Add _STATUS_KW_RE and _classify_status_kw at module level with full morphological coverage: продан/продана/продано, реализован[аоы]?, забронирован[аоы]?, свободн[аоы]?. Add 31 tests in test_domrf_catalog_parse.py covering all three extraction levels plus regression for fem. word-form «Квартира продана».	2026-06-17 20:40:49 +03:00
bot-backend	9dbfb03a3d	fix(scrapers): disambiguate EMISS yearly vs Q1 in dedup key (#1606) ... _YEAR_PERIODS ('год') → month=1 collided with Q1 ('I квартал') → month=1 for the same region/year: both produced obs_date=YYYY-01-01, so the second series in the XML silently overwrote the first in the in-memory by_key dict. Fix: add _emiss_period_granularity() and extend the dedup key from (region, obs_date) to (region, obs_date, granularity). 'year' and 'quarter' are now distinct slots, so both rows survive the parser and reach the upsert. Test: test_yearly_and_q1_both_survive_dedup verifies len==2 with both values present; test_period_granularity covers all classification branches.	2026-06-17 20:39:52 +03:00
bot-backend	14f3ef2019	fix(week-review): backend-аудит v2 — 82 фиксов (#1660) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-06-17 17:13:38 +00:00
Light1YT	86e9ea2937	fix(week-review): автофиксы код-ревью — 169 issue (label «week ревью 1») ... Многоагентный аудит + имплементация: один воркер на файл, точечные правки. Верификация: py_compile (47/47 .py) + tsc --noEmit (0 ошибок). Unit-тесты не прогонялись (окружение не поднято: rollup native dep / нет pytest-venv). Полностью исправлено (169): #1336, #1337, #1339, #1340, #1341, #1342, #1343, #1345, #1346, #1348, #1349, #1350, #1351, #1354, #1356, #1358, #1359, #1360, #1362, #1364, #1365, #1366, #1367, #1368, #1369, #1370, #1371, #1372, #1373, #1374, #1375, #1376, #1377, #1378, #1379, #1380, #1381, #1382, #1384, #1385, #1386, #1387, #1388, #1389, #1390, #1391, #1392, #1394, #1395, #1396, #1397, #1399, #1400, #1401, #1402, #1403, #1404, #1408, #1409, #1410, #1411, #1412, #1413, #1414, #1415, #1416, #1417, #1418, #1420, #1423, #1425, #1426, #1427, #1428, #1429, #1430, #1431, #1432, #1433, #1434, #1435, #1437, #1438, #1439, #1440, #1441, #1442, #1443, #1444, #1445, #1446, #1447, #1448, #1449, #1450, #1451, #1452, #1453, #1454, #1455, #1456, #1457, #1458, #1459, #1460, #1461, #1462, #1463, #1464, #1465, #1466, #1467, #1468, #1469, #1471, #1472, #1473, #1474, #1476, #1478, #1479, #1481, #1482, #1483, #1484, #1485, #1487, #1488, #1489, #1490, #1491, #1492, #1493, #1494, #1495, #1496, #1497, #1499, #1500, #1501, #1502, #1504, #1505, #1506, #1507, #1510, #1514, #1515, #1516, #1517, #1518, #1519, #1521, #1522, #1523, #1524, #1525, #1526, #1527, #1528, #1529, #1531, #1532, #1533, #1534, #1535, #1536, #1537, #1538 Частично (9, in-file часть, остаток cross-file): #1361, #1419, #1422, #1424, #1470, #1475, #1477, #1480, #1498 Требуют cross-file (3, не тронуты): #1338, #1363, #1421 Пропущено (1): #1539 Не входило в партию: 22 needs-Leha issue (нужны решения владельца). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 20:21:11 +05:00
Light1YT	eada9a25ac	fix(scrapers): izyatie OCR — '6б:' → '66:' для кад-номеров МСК-66 (#1228)	2026-06-13 15:03:59 +05:00
Light1YT	c3c6e2cef1	fix(okn): expand registry coverage + load detail-card name/address (#1159) ... okn_objects_sync собирал 428 ОКН из ~798 ЕГРОКН и сохранял raw_props={id} без названия/адреса — analyze показывал 'ОКН рядом' без указания КАКОГО. Coverage: расширил _ADDRESS_PASSES с 2 до 11 подстрок (Екатеринбург, г. Екатеринбург, 7 районов ЕКБ, 2 формы Свердловской обл.). _PASSES = декартово (address × category). first-wins дедуп сохраняет высшую категорию. Detail: fetch_okn_detail(source_id) — GET Show/Show?id=, BeautifulSoup парсит HTML (dl/dt/dd и tr/td), извлекает name, address, protection_category, object_kind, dating. _enrich_with_detail сливает в raw_props с graceful failover (cap=1500, exception per-object не валит прогон). Lookup: parcel_okn_objects возвращает name/address из raw_props->>'name'. Existing ird_analyze тесты не сломаны. 49 okn-related тестов passed, ruff clean. Closes #1159	2026-06-13 15:02:50 +05:00
Light1YT	285e8f974a	fix(sf): ekb_ppt_tep post-фильтры + врезка в analyze (#1136) ... Post-фильтры парсера (по образцу ppt2018_22823): - Табл.11: отбрасываем строки без zone_name (преамбульные артефакты merged-cells давали 2 фантома при 12 реальных зонах). - Табл.13: dedup по (phase, composition, zone, area_ha) + фильтры короткого-без-зоны и нумерационного шума (~364 строк → ~30-40 реальных). Analyze-врезка: новый ppt_tep_lookup.parcel_ppt_tep — JOIN planning_projects ⋈ ekb_ppt_tep по doc_ref↔source_key/doc_full_name (best-effort, без FK). Wired в build_ird_analyze_block рядом с planning_projects/krt_requisites — DB-источники, graceful. Seed-URL остаётся placeholder с # VERIFY (ingest пропускает с WARNING). Tests: 27 в parser (3 новых), 7 lookup, 22 wiring (2 новых). All ruff/syntax green. Closes #1136	2026-06-13 09:32:52 +00:00
Light1YT	e71224ef54	fix(scrapers): commit outer tx в domrf_catalog scrape_catalog_batch (#1227) ... begin_nested() лишь RELEASE'ит SAVEPOINT — без db.commit() все UPDATE цен откатывались при db.close() в Celery task. Зеркалит правильный паттерн scrape_catalog_objects (domrf_catalog_object.py:460): try- commit/except-rollback в конце функции. 50 catalog тестов зелёные. Closes #1227	2026-06-13 08:06:15 +00:00
Light1YT	5826812816	fix(domrf_kn): детерминированный sha256-fallback для flat_id (#1208) ... Fallback flat_id в _norm_flat использовал abs(hash(elem)) % (2**63 - 1). В CPython hash(str) РАНДОМИЗИРОВАН per-process (PYTHONHASHSEED нигде в репо не зафиксирован — uvicorn/celery не выставляют его). Эффект: - При resume упавшего sweep (resume_kn_run в новом процессе воркера) до 10 объектов после checkpoint перечитываются. Квартиры без flatId получают ДРУГОЙ hash-id → ON CONFLICT (id, snapshot_date) не срабатывает → дубли строк одной квартиры в одном snapshot. - То же при повторном прогоне за ту же дату — каждый раз новый id. - Дубли инфлируют все агрегации (units_sold/price медианы, supply_layers). Patch: - abs(hash(elem)) → int.from_bytes(sha256(elem)[:8], "big") % (2**63 - 1). - sha256 стабилен между процессами/перезапусками. 8 байт → BIGINT-fit. 6 новых юнит-тестов (test_domrf_kn_normalize.py): formula matches sha256, stable across calls, distinct elems→distinct ids, flatId wins over fallback, no-id→None, BIGINT-fit. ruff clean. Closes #1208	2026-06-13 06:09:54 +00:00
Light1YT	f9e045028c	feat(macro): land construction price index (СМР) via open Rosstat xlsx (#946) ... Closes the last #946 gap (СМР). Fetches «Индексы цен производителей на строительную продукцию по РФ» from the OPEN Rosstat xlsx (rosstat.gov.ru/storage/mediabank/Invest_ind_stroitel_MM-YYYY.xlsx), parses the monthly «к предыдущему месяцу» section → macro_indicator (indicator_type='construction_price_index', region='rf', monthly, %). The reCAPTCHA-gated fedstat dataGrid.do path is DELIBERATELY NOT used — the open Rosstat source needs no captcha. EMISS id=31108 documented as a prod-only SDMX alternative (not wired). URL is date-derived with bounded look-back (filename changes monthly; _fetch_construction_latest walks current month back to 6, first 200 wins; year-boundary handled, exhaustion raises, 404 breaks fast — no hang). Wired into rosstat_macro_sync with per-source try/except (can't break demography/income) + SAVEPOINT-per-row. None-not-0 on blank months. 33 tests (incl look-back + year-boundary + graceful), ruff clean. Note: no construction-cost channel in macro_coefficient yet (demand-side only) — series is ingested/available for future cost-side use + display. Refs #946	2026-06-09 11:28:05 +05:00
Light1YT	25e21c2bff	feat(macro): CBR inflation (ИПЦ YoY) -> macro_indicator + activate §9.5 channel (#946) ... fedstat ИПЦ is reCAPTCHA-blocked; CBR publishes inflation openly. Add fetch_inflation + parse_inflation_xlsx (CBR UniDbQuery DownloadExcel/132934, monthly % г/г, region=rf, source=cbr) to cbr_macro.py; upsert indicator_type=inflation_yoy via the existing cbr_macro_sync task (per-series guard, SAVEPOINT-per-row, CAST not ::, ON CONFLICT on the PK). Surface inflation_yoy in MonthlyMacro (frozen, carry-forward) and ACTIVATE the reserved §9.5 inflation channel (macro_coefficient f_inflation: level-vs-4%-target nudge, non-positive to avoid double-counting f_rate, excluded from _RATE_DRIVEN_FACTORS). Channel was DEGRADED (no data) -> now BACKED + consumed; _CONF_HIGH_MIN_BACKED 4->5. Deterministic (§16/§26); renorm claims the reserved 0.08 slice as designed. Live-verified (2026-04 5.58%); 194 macro + 902 forecasting tests green. No migration, no new deps. Refs #946.	2026-06-08 11:41:14 +05:00
bot-backend	5da4271235	feat(sf): OCR-пайплайн изъятия ЕКБ (Tesseract rus) → land_reservation (#1062) ... - Dockerfile: добавлены tesseract-ocr tesseract-ocr-rus в runner apt-слой - pyproject.toml: pymupdf>=1.24 (рендер PDF без poppler) + pytesseract>=0.3.13 - izyatie_client.py: list_izyatie_documents() + fetch_pdf() с екатеринбург.рф - izyatie_ocr.py: ocr_pdf_text() (PyMuPDF→PIL→Tesseract rus) + extract_izyatie_records() с нормализацией OCR-шума (пробелы в кад-номерах, кирилл. О→0, б→6) - izyatie_ocr_ingest.py: Celery task → land_reservation UPSERT (SAVEPOINT per-row) - beat_schedule.py: izyatie-ocr-ingest-weekly (пятница 07:00 МСК) - 33 теста, все зелёные; без реального Tesseract/сети в CI	2026-06-07 18:07:06 +03:00
bot-backend	e2faab801c	fix(sf): pravo.gov66 law_type числовой код =1 Постановление (#1149) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-06-07 14:17:52 +00:00
bot-backend	a6e100b52a	fix(sf): корректное urlencode form-data в okn_egrkn_client (#1147) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-06-07 14:13:34 +00:00
bot-backend	9ee35bebf0	fix(sf): User-Agent header for okn-mk searchMap (bypass 403) (#1146) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-06-07 13:57:34 +00:00
lekss361	396db20c48	feat(sf): enumerate изъятия/резервирования с pravo.gov66 → land_reservation (#1145) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 13:56:29 +00:00
Light1YT	1f66dfd400	feat(macro): ЕМИСС среднедушевые доходы (id=57039) -> macro_indicator (#946 part2) ... Расширяет Росстат-скрейпер ЕМИСС/fedstat-рядом «Среднедушевые денежные доходы населения» (fedstat id=57039, OKATO 65=Свердл, квартальный, руб). Добавляет pure SDMX-ML v1.0 GenericData-парсер (stdlib ElementTree, dev-тестируем на фикстурах) + EMISS-ветку Celery-таски rosstat_macro_sync (open-data + ЕМИСС, per-source guard, source='emiss', frequency параметризован, CAST not ::, SAVEPOINT per-row). income_per_capita проброшен в site_finder/macro (region='sverdl'). ЕМИСС за WAF с dev -> fetch исполняется на проде (verified: httpx POST из gendesign-worker-1 -> 200 + SDMX, парсер извлёк 9 Свердл-строк). ИПЦ (id=31074) / индекс цен СМР НЕ landed: многомерные, дефолт-экспорт data.do без Свердл, нужна dataGrid-выборка (selectedFilterIds + filter-tree AJAX + reCAPTCHA) -- задокументировано. Refs #946.	2026-06-07 18:44:38 +05:00
Light1YT	f7c4d7a8c5	feat(macro): Росстат open-data macro scraper -> macro_indicator (#946) ... EPIC2 macro-ingest: региональный (Свердловская обл. ОКТМО 65 / ЕКБ) скрейпер Росстата в унифицированную macro_indicator (м.123). Зеркалит форму cbr_macro: pure-парсеры + тонкий httpx, Celery-таска с SAVEPOINT-per-row upsert (CAST not ::, ON CONFLICT по PK indicator_type/region/obs_date). ЕМИСС/fedstat.ru (SDMX) за WAF (hard-403 на всех путях из dev-IP) -> ИПЦ/доходы/ СМР-цены пока недоступны (документировано в коде). rosstat.gov.ru/opendata (стандарт 4.0) WAF-free -> приземляет демографию population_total (sverdl 4.32M / ekb 1.41M, §7.11). - services/scrapers/rosstat_emiss.py — fetch+parse open-data (registry->meta->data CSV) - workers/tasks/rosstat_macro_sync.py — Celery upsert (source=rosstat) - beat: rosstat-macro-sync-monthly; include в celery_app - site_finder/macro.py: population_total -> region-aware default sverdl (additive) - tests: 18 offline-тестов парсера + контракта таски Без миграции (вписано в существующую схему), без новых зависимостей. Refs #946.	2026-06-07 13:12:01 +00:00
lekss361	50fcba1ca0	feat(sf): ingest ОКН-объектов ЕКБ из АИС ЕГРКН (точки+категория) → okn_objects (#1141) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 12:51:18 +00:00
lekss361	189ecc81f8	feat(sf): ingest функц.зон генплана ЕКБ-2045 из ГИСОГД-СО WFS → ekb_genplan_functional_zone (#1137) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 12:03:39 +00:00
lekss361	1e53416277	feat(sf): ППТ/ПМТ ТЭП-парсер (Табл.11/12/13) → ekb_ppt_tep MVP (#1133) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 11:54:40 +00:00
lekss361	57c6d119f5	feat(sf): scraper реестра КРТ-площадок ЕКБ → ekb_krt_sites (#1126) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 11:24:49 +00:00
lekss361	ef15190690	feat(sf): geocode geom для ekburg_construction_permits из МСК-66 (#1124) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 11:16:45 +00:00
lekss361	59f2628e0b	feat(sf): ПАГЕ-парсер изъятия/резервирования → land_reservation (#1118) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 10:15:03 +00:00
lekss361	dcafb32f31	feat(sf): gknspecial_zone ИРД-harvest → ird_overlays (#1114) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-07 09:54:53 +00:00
lekss361	e7a0af5ddc	fix(sf): ekb_geoportal_client verify=False — росс-гос-CA не в trust-store контейнера (#1103) ... _http_get_json/_http_post_json → verify=False (const _VERIFY_TLS): геопортал ЕКБ отдаёт цепочку с росс-гос-CA, которого нет в trust-store контейнера → verify=True падал CERTIFICATE_VERIFY_FAILED, весь geoportal-слой (C8b/D9b/#1085) не работал бы с прода. Данные публичные open-data, секреты не передаются. Прецедент: nspd_lite/ekburg_permits/pzz_loader. Прод-probe подтвердил. +регрессия-тест. Refs #1067. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 20:10:26 +00:00
lekss361	b1530467f7	feat(sf): opportunity-ЗУ harvest → ird_overlays (future-supply, #1100) ... opportunity_harvest.py — grid-walk НСПД opportunity-слоёв (ЗУ аукцион/схема/свободные/межевание/ККР) по кварталам ЕКБ → ird_overlays с layer_kind='opportunity_*'. Переиспользует движок ird_harvest (#1090): _upsert_feature/_ekb_quarters. +krt_territories=37430 в LAYERS. Beat monthly (1-е 05:30). Self-contained, прод-прогон gated B6. Closes #1086. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 19:32:45 +00:00
lekss361	b4e6e43713	feat(sf): EKB geoportal urbanCard — ПЗЗ-регламент content (#1094) ... Расширение ekb_geoportal_client.py — urbanCard pipeline для ПЗЗ-регламента (#1067 C8): searchByGeom → urbanCard → parse_zone_regulation → ZoneRegulation (zone_index + 3 списка ВРИ 540н + предельные параметры). Метод zone_regulation_at(lon,lat). Supplement к WFS (C7), аддитивно. 11 тестов. Wiring в C8b cache-таблицу. Refs #1067, #1059. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 19:05:47 +00:00
lekss361	4e41f9be73	feat(sf): EKB geoportal WFS client (#1083) ... Новый ekb_geoportal_client.py — WFS-клиент муниципального геопортала ЕКБ (Geometa GeoServer, public). features_at_point (CQL INTERSECTS), features_in_bbox (CQL BBOX), zone_index_at (crosswalk координата→индекс ПЗЗ). EPSG:4326, OGC WFS GetFeature (не WMS — reproject-баг). Единый источник землепользование/КРТ/ППТ/ЗОУИТ для ЕКБ. 7 тестов. Foundation #1067 C7; wiring в C8/D9. Refs #1067. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 18:37:09 +00:00
lekss361	c2e0428803	feat(nspd): get_feature_info FEATURE_COUNT=10 — multi-feature per point (#1080) ... get_feature_info() шлёт WMS FEATURE_COUNT=10 (новый параметр feature_count, default 10). NSPD по умолчанию отдаёт 1 фичу/точку → перекрывающиеся зоны одного слоя теряются (37581 «иные ЗОУИТ» на ЕКБ = 2 фичи). Tile-геометрия не тронута. +2 теста (multi-feature parse, override), существующий обновлён. Refs #1067. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 18:18:07 +00:00
lekss361	38d948f078	fix(nspd): list_layers parse flat layers[] response, not children-tree (#1077) ... NSPDClient.list_layers() парсил _walk_layer_tree по id/children, но НСПД layers-theme-tree отдаёт слои в плоском layers[] с ключом layerId → list_layers() молча возвращал [] для любой темы. Fix: сначала плоский data["layers"], иначе fallback на _walk_layer_tree; defensive None-guard на missing id. +тест на flat-форму, backward-compat сохранён. Refs #1067. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 17:51:14 +00:00
Light1YT	420cda2831	feat(domrf): persist domrf_kn_objects.free_flats (#950 PR A) (#1002)	2026-06-02 20:06:29 +00:00
Light1YT	c78c6ec2d0	feat(site-finder): live CBR key-rate scraper → macro_indicator (#945 PR B) (#964)	2026-06-02 19:05:57 +00:00
Light1YT	4e4762f234	fix(tradein): backfill kn obj_class (parser + 3-pass) (#572) (#628)	2026-05-28 15:04:32 +00:00
Light1YT	609ac8ea9d	fix(tradein): backfill kn price_per_m2 via sale_graph + peer median (#573) (#625)	2026-05-28 13:50:45 +00:00
lekss361	099b8e4b59	fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503)	2026-05-24 10:53:28 +00:00
lekss361	9ffeee7ed2	feat(22d): admin UI buttons (skip/force) + reduce catalog jitter 800→300ms (#352)	2026-05-17 23:38:22 +00:00
lekss361	e5498682ba	fix(22d): commit outer transaction in scrape_catalog_objects orchestrator ... PR #335 использовал `with db.begin_nested():` SAVEPOINT для per-row UPDATE, но outer transaction никогда не commit'ился. После цикла Celery task закрывал session (`db.close()` в finally) → autobegin'd outer tx rollback → все UPDATE'ы теряются. Симптом: task завершается с processed=3 succeeded=3, в Glitchtip нет ошибок, но в БД ни одного catalog_scraped_at != NULL. Fix: добавлен db.commit() после цикла + rollback на ошибку commit'а.	2026-05-18 01:10:06 +03:00
lekss361	0567ad2130	feat(22d): domrf_catalog_object scraper — fill ~25 NULL kn_objects cols from SSR __NEXT_DATA__ (#335)	2026-05-17 21:26:22 +00:00
lekss361	2aa33e0eb8	fix(stealth): bump Semaphore 6->8 for higher concurrency in gather ... Per-object 7 endpoints через asyncio.gather упираются в semaphore limit. 6 -> 8 — ещё +33% concurrency для batch fetch'а.	2026-05-17 23:04:51 +03:00
lekss361	0f289c05b1	fix(scrape-kn): real documents endpoints (5x) + skip obj_checks 404 + sem 3->6 ... PR #322 used placeholder URLs: - /api/object/{id}/documents -> 404 (Playwright showed: 5 real endpoints: document/rpd / developer/report / project/documentation / documentation/other / document/permits). Replaced with 5 parallel fetches + merge into single upsert_documents call. - /api/object/{id}/checks -> 404. 6 checkboxes likely inline in bulk kn/object payload; endpoint /checks does not exist. Commented out _fetch_obj_checks_safe with TODO for separate investigation PR. - BrowserSession Semaphore(3) -> Semaphore(6) — double concurrency for intra-object gather. GlitchTip: kn_scrape_failures run #19 showed 40 failures each for both placeholder endpoints across first 30 objects of 1532.	2026-05-17 22:59:44 +03:00
lekss361	f2983c86d7	feat(scrape-kn): parallel Phase B/C + wire obj_checks + documents + is_ekb auto-derive (#322)	2026-05-17 19:40:16 +00:00