gendesign/backend/app/services/scrapers
lekss361 1ebefcdb98 feat(22d): domrf_catalog.py — per-flat price/status from SSR HTML catalog page
Adds new scraper module for DOM.RF catalog-квартир SSR pages
(issue #297 Phase 5). kn-API returns price for only 0.3% of EKB flats;
prices + finishing/ceiling/section data live on per-flat HTML pages at
/сервисы/каталог-квартир/квартира/{hash}.

- fetch_catalog_html(): in-browser fetch via BrowserSession (same WAF
  bypass pattern as get_json, but returns text/html instead of JSON)
- parse_catalog_flat(): 2-level extraction via stdlib html.parser +
  regex on raw HTML (no new deps; beautifulsoup4 noted as TODO)
- upsert_catalog_data(): UPDATE domrf_kn_flats SET catalog-only cols
  with COALESCE guards; SAVEPOINT per row; does not touch kn-API metadata
- scrape_catalog_batch(): full batch orchestrator over one BrowserSession

Wiring (Celery task + beat schedule) deferred to a follow-up PR.
Depends on migration 22b (section_no, finishing_type, ceiling_height_m,
catalog_updated_at, catalog_url_hash columns) — also separate PR.

Closes #297 (partial — 22d code delivery)
2026-05-17 18:34:10 +03:00
..
__init__.py add interactive parser of наш.дом.рф ЖК+flats with stealth Playwright 2026-04-27 17:58:40 +03:00
documents.py feat(22i): domrf_kn_documents new table + parser (#305) 2026-05-17 15:21:31 +00:00
domrf_catalog.py feat(22d): domrf_catalog.py — per-flat price/status from SSR HTML catalog page 2026-05-17 18:34:10 +03:00
domrf_kn.py feat(22begh): schema extension — flats+6 / objects+30 / developers+4 / metro_top3 (#306) 2026-05-17 15:26:59 +00:00
ekburg_permits.py fix(#242): ekburg_permits SSL — verify=False для CA Минцифры РФ 2026-05-17 08:57:42 +03:00
nspd_client.py fix(#231): repair test_rate_limit_semaphore regression + docstring sync 2026-05-17 10:22:16 +03:00
nspd_denorm.py feat(#126): Sub-PR A — NSPDClient.get_features_in_bbox_grid + engineering classifier (#247) 2026-05-17 05:54:30 +00:00
nspd_lite.py fix(nspd-geo): rosreestr2coord v5 compat — delay arg, tmp cache, dumps default 2026-05-11 14:43:44 +03:00
objective.py feat(objective): integrate api.objctv.ru — auth, schema, weekly sync 2026-05-07 21:04:00 +03:00
stealth.py add domrf extras parser (sale_graph + sales_agg + infra + photos) + 2026-04-27 18:35:46 +03:00