|
|
ef0b0d0b3a
|
feat(scheduler): add dormant Yandex Realty city-sweep (#561)
Wire a full Yandex.Недвижимость periodic city-sweep into the in-app
scheduler, shipped DISABLED (enabled=false) by design — live behavior is
intentionally unverified pending egress-IP rotation (#623). Mirrors the
existing avito_city_sweep (also currently disabled in prod).
- run_yandex_city_sweep in scrape_pipeline.py: mirrors run_avito_city_sweep
but simpler (no houses/detail enrich) — search + save_listings per anchor,
cooperative cancel, per-anchor exception isolation, defensive abort after
3 consecutive anchor failures (mark_banned; YandexRealtyScraper has no
block/rate exception classes — fetch_around swallows errors → []).
- trigger_yandex_city_sweep_run + dispatch branch in scheduler.py (faithful
mirror of the avito trigger: claim run, fresh SessionLocal, create_task).
- Migration 078 seeds scrape_schedules row enabled=false, window 02:00-05:00
UTC, conservative params {pages_per_anchor:2, request_delay_sec:9,
radius_m:1500}, ON CONFLICT (source) DO NOTHING (idempotent).
- 8 unit tests (mocked scraper + save_listings, no live network/DB):
anchor iteration, cancel mid-sweep, per-anchor isolation, consecutive-
failure abort + reset-on-success, empty-page pagination break.
Enable later (after #623): UPDATE scrape_schedules SET enabled=true
WHERE source='yandex_city_sweep';
Closes #561
|
2026-05-29 13:49:28 +05:00 |
|
