gendesign/backend/app/services/scrapers
Light1YT e961ea9435 feat(domrf): persist domrf_kn_objects.free_flats (#950 PR A)
Parse freeFlatsInfo.numberFlats (currently-listed flat count) from the
kn-API payload into a new nullable free_flats INT, mirroring price_min_rub
(extraction via _to_int, UPSERT column/bind, ON CONFLICT overwrite). _to_int
rejects bool, preserves 0, truncates float, and is overflow-safe.

Migration 124 adds the column + COMMENT and backfills historical rows
deterministically from the stored raw payload (domrf_raw_endpoints, section
kn_api) without re-scraping — same jsonb_array_elements pattern already
proven on prod in migration 121, plus a {1,9}/{1,18} digit-cap so a stray
numeric string can never overflow int4/bigint at the deploy gate.

Unblocks object-level Layer2 (Site Finder v2 EPIC 6): hidden = flat_count
(total PD) − free_flats (listed now).
2026-06-03 01:05:05 +05:00
..
__init__.py add interactive parser of наш.дом.рф ЖК+flats with stealth Playwright 2026-04-27 17:58:40 +03:00
cbr_macro.py feat(site-finder): live CBR key-rate scraper → macro_indicator (#945 PR B) (#964) 2026-06-02 19:05:57 +00:00
documents.py feat(22i): domrf_kn_documents new table + parser (#305) 2026-05-17 15:21:31 +00:00
domrf_catalog.py feat(22d): domrf_catalog.py scraper for per-flat price/status from SSR HTML (#309) 2026-05-17 15:40:01 +00:00
domrf_catalog_object.py fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503) 2026-05-24 10:53:28 +00:00
domrf_kn.py feat(domrf): persist domrf_kn_objects.free_flats (#950 PR A) 2026-06-03 01:05:05 +05:00
ekburg_permits.py fix(#242): ekburg_permits SSL — verify=False для CA Минцифры РФ 2026-05-17 08:57:42 +03:00
flat_plans.py feat(22c): domrf_kn_flat_plans new table + parser stub (#308) 2026-05-17 15:39:34 +00:00
nspd_client.py fix(#231): repair test_rate_limit_semaphore regression + docstring sync 2026-05-17 10:22:16 +03:00
nspd_denorm.py feat(#126): Sub-PR A — NSPDClient.get_features_in_bbox_grid + engineering classifier (#247) 2026-05-17 05:54:30 +00:00
nspd_lite.py fix(nspd-geo): rosreestr2coord v5 compat — delay arg, tmp cache, dumps default 2026-05-11 14:43:44 +03:00
obj_checks.py feat(scrape-kn): parallel Phase B/C + wire obj_checks + documents + is_ekb auto-derive (#322) 2026-05-17 19:40:16 +00:00
objective.py fix(scrape-objective): stream-parse lots_pf via ijson to avoid OOM (#315) 2026-05-17 16:37:30 +00:00
stealth.py fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503) 2026-05-24 10:53:28 +00:00