feat(domclick): wire DomClick citywide sweep into pipeline+scheduler (seeded disabled) #1730

Merged

lekss361 merged 1 commit from feat/activate-domclick-sweep into main

2026-06-17 20:29:44 +00:00

lekss361 commented

2026-06-17 20:24:30 +00:00

Owner

Summary

Activates the existing-but-unwired DomClickScraper (#796) — a complete camoufox/BrowserFetcher scraper that was never connected to the pipeline (0 rows, no schedule). Validated live (home IP): EKB city_id=4, SERP returns 40 /card/sale__flat__<id> cards, "2 398" 2-room listings, not DataDome-blocked via browser.

Changes

scrape_pipeline.py: run_domclick_city_sweep + DomClickCitySweepCounters — CITYWIDE (no geo anchors; SERP has no coords). fetch_city(city_id, rooms, pages) → save_listings (triggers house-match hook; fingerprint-tier only since lat/lon NULL). Same asyncio.wait_for watchdog pattern as siblings.
scheduler.py: dispatch domclick_city_sweep → run_domclick_city_sweep + trigger_domclick_city_sweep_run (mirrors yandex sibling), passes default_params.
data/sql/122_enable_domclick_city_sweep.sql: idempotent seed, enabled=false (dormant — enable after manual prod-smoke), window 3-6, params {city_id:4, rooms:[0..4], pages_per_anchor:5, request_delay_sec:6}.

⚠️ Scraper's canonical source string is "domklik" (pre-existing in domclick.py, left untouched) — listings will have source='domklik'.

Test plan

pytest -k "domclick or pipeline or sweep or scheduler" → 163 passed; ruff clean
sweep saves lots + counters; _parse_html card→lot
Post-deploy MANUAL smoke: enable schedule + next_run=now → verify lots inserted, not banned → then set production window

Refs #796

## Summary Activates the existing-but-unwired `DomClickScraper` (#796) — a complete camoufox/BrowserFetcher scraper that was never connected to the pipeline (0 rows, no schedule). Validated live (home IP): EKB `city_id=4`, SERP returns 40 `/card/sale__flat__<id>` cards, "2 398" 2-room listings, not DataDome-blocked via browser. ## Changes - `scrape_pipeline.py`: `run_domclick_city_sweep` + `DomClickCitySweepCounters` — CITYWIDE (no geo anchors; SERP has no coords). `fetch_city(city_id, rooms, pages)` → `save_listings` (triggers house-match hook; fingerprint-tier only since lat/lon NULL). Same `asyncio.wait_for` watchdog pattern as siblings. - `scheduler.py`: dispatch `domclick_city_sweep` → `run_domclick_city_sweep` + `trigger_domclick_city_sweep_run` (mirrors yandex sibling), passes default_params. - `data/sql/122_enable_domclick_city_sweep.sql`: idempotent seed, **`enabled=false`** (dormant — enable after manual prod-smoke), window 3-6, params `{city_id:4, rooms:[0..4], pages_per_anchor:5, request_delay_sec:6}`. ⚠️ Scraper's canonical source string is `"domklik"` (pre-existing in domclick.py, left untouched) — listings will have `source='domklik'`. ## Test plan - [x] `pytest -k "domclick or pipeline or sweep or scheduler"` → 163 passed; ruff clean - [x] sweep saves lots + counters; `_parse_html` card→lot - [ ] Post-deploy MANUAL smoke: enable schedule + next_run=now → verify lots inserted, not banned → then set production window Refs #796

lekss361 added 1 commit 2026-06-17 20:24:30 +00:00