gendesign/backend/app/services/scrapers
lekss361 54cff64109 fix(bug-22a): fill ON CONFLICT gaps for flat_number/living_area/wall_type/energy_eff in kn scraper
Audit of kn-API payload for issue #297 sub-task 22a:

Object-level (/kn/object list endpoint):
- wallType, energyEff, objClass: absent from API payload (0/1516 objects).
  obj_class already fixed in SF#15 via aiDescription regex.
  wall_type/energy_eff: no data source exists.

Flat-level (/portal-kn/api/sales/portal/table):
- number → flat_number: mapped correctly in _norm_flat().
- livingArea → living_area: mapped correctly in _norm_flat().
- Root cause for 0% DB coverage: UPSERT_FLAT_SQL ON CONFLICT did not include
  flat_number, living_area, is_studio, total_area, rooms, floor, num_floors —
  existing rows (hash-id, inserted before direct flatId became available)
  were never updated on subsequent scrapes.

Fix:
- UPSERT_FLAT_SQL ON CONFLICT: add COALESCE updates for all 7 flat fields.
- UPSERT_OBJECT_SQL ON CONFLICT: add COALESCE for wall_type + energy_eff
  (obj_class was already updating, now also COALESCE to preserve existing value).
- Add migration 108 (idempotent DDL guard + coverage NOTICE).

Coverage will improve on next scrape; no backfill possible (flat raw not stored).

Closes #297 sub-task 22a.
2026-05-17 17:54:18 +03:00
..
__init__.py add interactive parser of наш.дом.рф ЖК+flats with stealth Playwright 2026-04-27 17:58:40 +03:00
domrf_kn.py fix(bug-22a): fill ON CONFLICT gaps for flat_number/living_area/wall_type/energy_eff in kn scraper 2026-05-17 17:54:18 +03:00
ekburg_permits.py fix(#242): ekburg_permits SSL — verify=False для CA Минцифры РФ 2026-05-17 08:57:42 +03:00
nspd_client.py fix(#231): repair test_rate_limit_semaphore regression + docstring sync 2026-05-17 10:22:16 +03:00
nspd_denorm.py feat(#126): Sub-PR A — NSPDClient.get_features_in_bbox_grid + engineering classifier (#247) 2026-05-17 05:54:30 +00:00
nspd_lite.py fix(nspd-geo): rosreestr2coord v5 compat — delay arg, tmp cache, dumps default 2026-05-11 14:43:44 +03:00
objective.py feat(objective): integrate api.objctv.ru — auth, schema, weekly sync 2026-05-07 21:04:00 +03:00
stealth.py add domrf extras parser (sale_graph + sales_agg + infra + photos) + 2026-04-27 18:35:46 +03:00