1ebefcdb98
Adds new scraper module for DOM.RF catalog-квартир SSR pages (issue #297 Phase 5). kn-API returns price for only 0.3% of EKB flats; prices + finishing/ceiling/section data live on per-flat HTML pages at /сервисы/каталог-квартир/квартира/{hash}. - fetch_catalog_html(): in-browser fetch via BrowserSession (same WAF bypass pattern as get_json, but returns text/html instead of JSON) - parse_catalog_flat(): 2-level extraction via stdlib html.parser + regex on raw HTML (no new deps; beautifulsoup4 noted as TODO) - upsert_catalog_data(): UPDATE domrf_kn_flats SET catalog-only cols with COALESCE guards; SAVEPOINT per row; does not touch kn-API metadata - scrape_catalog_batch(): full batch orchestrator over one BrowserSession Wiring (Celery task + beat schedule) deferred to a follow-up PR. Depends on migration 22b (section_no, finishing_type, ceiling_height_m, catalog_updated_at, catalog_url_hash columns) — also separate PR. Closes #297 (partial — 22d code delivery) |
||
|---|---|---|
| .. | ||
| cadastre | ||
| etl | ||
| exporters | ||
| generative | ||
| photos | ||
| scrapers | ||
| site_finder | ||
| __init__.py | ||
| analytics_queries.py | ||
| analytics_refresh.py | ||
| job_settings.py | ||
| objective_etl.py | ||
| objective_sync_config.py | ||