feat(22d): domrf_catalog.py scraper for per-flat price/status from SSR HTML #309

Merged
lekss361 merged 1 commit from fix/22d-catalog-scraper into main 2026-05-17 15:40:02 +00:00

1 commit

Author SHA1 Message Date
lekss361
1ebefcdb98 feat(22d): domrf_catalog.py — per-flat price/status from SSR HTML catalog page
Adds new scraper module for DOM.RF catalog-квартир SSR pages
(issue #297 Phase 5). kn-API returns price for only 0.3% of EKB flats;
prices + finishing/ceiling/section data live on per-flat HTML pages at
/сервисы/каталог-квартир/квартира/{hash}.

- fetch_catalog_html(): in-browser fetch via BrowserSession (same WAF
  bypass pattern as get_json, but returns text/html instead of JSON)
- parse_catalog_flat(): 2-level extraction via stdlib html.parser +
  regex on raw HTML (no new deps; beautifulsoup4 noted as TODO)
- upsert_catalog_data(): UPDATE domrf_kn_flats SET catalog-only cols
  with COALESCE guards; SAVEPOINT per row; does not touch kn-API metadata
- scrape_catalog_batch(): full batch orchestrator over one BrowserSession

Wiring (Celery task + beat schedule) deferred to a follow-up PR.
Depends on migration 22b (section_no, finishing_type, ceiling_height_m,
catalog_updated_at, catalog_url_hash columns) — also separate PR.

Closes #297 (partial — 22d code delivery)
2026-05-17 18:34:10 +03:00