gendesign/backend
lekss361 1ebefcdb98 feat(22d): domrf_catalog.py — per-flat price/status from SSR HTML catalog page
Adds new scraper module for DOM.RF catalog-квартир SSR pages
(issue #297 Phase 5). kn-API returns price for only 0.3% of EKB flats;
prices + finishing/ceiling/section data live on per-flat HTML pages at
/сервисы/каталог-квартир/квартира/{hash}.

- fetch_catalog_html(): in-browser fetch via BrowserSession (same WAF
  bypass pattern as get_json, but returns text/html instead of JSON)
- parse_catalog_flat(): 2-level extraction via stdlib html.parser +
  regex on raw HTML (no new deps; beautifulsoup4 noted as TODO)
- upsert_catalog_data(): UPDATE domrf_kn_flats SET catalog-only cols
  with COALESCE guards; SAVEPOINT per row; does not touch kn-API metadata
- scrape_catalog_batch(): full batch orchestrator over one BrowserSession

Wiring (Celery task + beat schedule) deferred to a follow-up PR.
Depends on migration 22b (section_no, finishing_type, ceiling_height_m,
catalog_updated_at, catalog_url_hash columns) — also separate PR.

Closes #297 (partial — 22d code delivery)
2026-05-17 18:34:10 +03:00
..
alembic ops: alembic baseline, pre-commit, TIGER cleanup, pg_dump scripts 2026-04-26 13:08:51 +03:00
app feat(22d): domrf_catalog.py — per-flat price/status from SSR HTML catalog page 2026-05-17 18:34:10 +03:00
db/init ops: alembic baseline, pre-commit, TIGER cleanup, pg_dump scripts 2026-04-26 13:08:51 +03:00
output feat(geo): NSPD bulk-fetcher без Playwright + resume-friendly UI 2026-05-11 08:53:28 +03:00
tests fix(sf-09): capacity-aware redistribute (round 2) — bounded + correct invariant 2026-05-17 15:30:50 +03:00
.dockerignore fix(scraper): диагностируем «кнопка не работает» — Redis lock + task_received log 2026-04-28 23:17:42 +03:00
.env.example feat(backend): sentry-sdk init для FastAPI + Celery (#204 backend) (#207) 2026-05-16 15:09:16 +00:00
.env.runtime.example sprint1: nspd scraper industrialization, per-bucket elasticity, cadastre cross-check, sentry releases 2026-04-30 21:51:19 +03:00
alembic.ini ops: alembic baseline, pre-commit, TIGER cleanup, pg_dump scripts 2026-04-26 13:08:51 +03:00
debug.log feat(geo): NSPD bulk-fetcher без Playwright + resume-friendly UI 2026-05-11 08:53:28 +03:00
Dockerfile fix(scraper): диагностируем «кнопка не работает» — Redis lock + task_received log 2026-04-28 23:17:42 +03:00
pyproject.toml feat(backend): sentry-sdk init для FastAPI + Celery (#204 backend) (#207) 2026-05-16 15:09:16 +00:00
uv.lock feat(backend): sentry-sdk init для FastAPI + Celery (#204 backend) (#207) 2026-05-16 15:09:16 +00:00