gendesign/backend/app/services/scrapers
Light1YT dfae54384e fix(domrf): backfill kn obj_class via aiDescription + older snapshots (#572)
kn-API /kn/object endpoint never populates objClass (0/1516 objects confirmed
in bug-22a audit). Parser _extract_obj_class_from_ai() worked but lacked the
"типов" pattern, missing "типового класса"/"типовой серии" in 131+ objects.
The 2026-05-24 sweep was also truncated (1 object in raw payload) leaving 620
new rows with no aiDescription to extract from.

Fix:
- Add (r"типов", "Типовой") to _OBJ_CLASS_PATTERNS in domrf_kn.py; aligns with
  the "Типовой" value that domrf_catalog.py (SSR scraper) already sets from pages.
- Migration 120: three-pass backfill covering all snapshots:
  Pass 1: aiDescription from all stored raw payloads with extended pattern
  Pass 2: propagate obj_class forward from most recent older snapshot (covers
          broken 2026-05-24 sweep — 185 objects)
  Pass 3: objective_complex_mapping → objective_lots modal class (~14 objects)
  Projects 50.2% → 71.2% coverage (3819→5418/7614 rows); meets ≥70% acceptance.
2026-05-28 18:48:31 +05:00
..
__init__.py add interactive parser of наш.дом.рф ЖК+flats with stealth Playwright 2026-04-27 17:58:40 +03:00
documents.py feat(22i): domrf_kn_documents new table + parser (#305) 2026-05-17 15:21:31 +00:00
domrf_catalog.py feat(22d): domrf_catalog.py scraper for per-flat price/status from SSR HTML (#309) 2026-05-17 15:40:01 +00:00
domrf_catalog_object.py fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503) 2026-05-24 10:53:28 +00:00
domrf_kn.py fix(domrf): backfill kn obj_class via aiDescription + older snapshots (#572) 2026-05-28 18:48:31 +05:00
ekburg_permits.py fix(#242): ekburg_permits SSL — verify=False для CA Минцифры РФ 2026-05-17 08:57:42 +03:00
flat_plans.py feat(22c): domrf_kn_flat_plans new table + parser stub (#308) 2026-05-17 15:39:34 +00:00
nspd_client.py fix(#231): repair test_rate_limit_semaphore regression + docstring sync 2026-05-17 10:22:16 +03:00
nspd_denorm.py feat(#126): Sub-PR A — NSPDClient.get_features_in_bbox_grid + engineering classifier (#247) 2026-05-17 05:54:30 +00:00
nspd_lite.py fix(nspd-geo): rosreestr2coord v5 compat — delay arg, tmp cache, dumps default 2026-05-11 14:43:44 +03:00
obj_checks.py feat(scrape-kn): parallel Phase B/C + wire obj_checks + documents + is_ekb auto-derive (#322) 2026-05-17 19:40:16 +00:00
objective.py fix(scrape-objective): stream-parse lots_pf via ijson to avoid OOM (#315) 2026-05-17 16:37:30 +00:00
stealth.py fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503) 2026-05-24 10:53:28 +00:00