gendesign/backend/app/services/scrapers
Light1YT 57214f7c49 fix(tradein): backfill kn price_per_m2 (#573)
756 977 NULL-status rows in domrf_kn_flats imported from historical
DOM.RF snapshots carry total_area but no price_rub, so the scraper's
price_rub/total_area derivation never fires for them. This left 5/8 EKB
districts at 0% price_per_m2 coverage and blocked the
Limit_District_Median_Partial analytics limitation.

Migration 100 adds a three-pass idempotent backfill:
  1. price_rub / total_area (sanity net; 0 rows in practice)
  2. domrf_kn_sale_graph.price_avg (type='apartments', latest snapshot)
     via obj_id join — fills 453 606 rows
  3. peer median from free/sold/booked flats in the same obj_id — fills
     620 more rows

After backfill: 61.5% overall coverage (up from 3.7%), all 8 EKB
districts get a non-NULL median (7/8 districts reach ≥ 60%).

Also tightens the UPSERT: price_per_m2 now uses COALESCE so re-scraping
NULL-status flats no longer overwrites a previously backfilled value.
2026-05-28 18:31:50 +05:00
..
__init__.py add interactive parser of наш.дом.рф ЖК+flats with stealth Playwright 2026-04-27 17:58:40 +03:00
documents.py feat(22i): domrf_kn_documents new table + parser (#305) 2026-05-17 15:21:31 +00:00
domrf_catalog.py feat(22d): domrf_catalog.py scraper for per-flat price/status from SSR HTML (#309) 2026-05-17 15:40:01 +00:00
domrf_catalog_object.py fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503) 2026-05-24 10:53:28 +00:00
domrf_kn.py fix(tradein): backfill kn price_per_m2 (#573) 2026-05-28 18:31:50 +05:00
ekburg_permits.py fix(#242): ekburg_permits SSL — verify=False для CA Минцифры РФ 2026-05-17 08:57:42 +03:00
flat_plans.py feat(22c): domrf_kn_flat_plans new table + parser stub (#308) 2026-05-17 15:39:34 +00:00
nspd_client.py fix(#231): repair test_rate_limit_semaphore regression + docstring sync 2026-05-17 10:22:16 +03:00
nspd_denorm.py feat(#126): Sub-PR A — NSPDClient.get_features_in_bbox_grid + engineering classifier (#247) 2026-05-17 05:54:30 +00:00
nspd_lite.py fix(nspd-geo): rosreestr2coord v5 compat — delay arg, tmp cache, dumps default 2026-05-11 14:43:44 +03:00
obj_checks.py feat(scrape-kn): parallel Phase B/C + wire obj_checks + documents + is_ekb auto-derive (#322) 2026-05-17 19:40:16 +00:00
objective.py fix(scrape-objective): stream-parse lots_pf via ijson to avoid OOM (#315) 2026-05-17 16:37:30 +00:00
stealth.py fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503) 2026-05-24 10:53:28 +00:00