feat(scheduler): dormant Yandex Realty city-sweep (#561) #642

Merged

Light1YT merged 1 commit from feat/561-yandex-sweep-scheduler into main

2026-05-29 08:50:10 +00:00

Light1YT commented

2026-05-29 08:49:57 +00:00

Owner

Summary

Closes #561. Wires a full Yandex.Недвижимость periodic city-sweep into the in-app scheduler, shipped DISABLED (enabled=false) by design — live behavior is intentionally unverified pending egress-IP rotation (#623). Mirrors the existing avito_city_sweep (also currently disabled in prod).

run_yandex_city_sweep (scrape_pipeline.py): mirror of run_avito_city_sweep, simpler (no houses/detail enrich) — search + save_listings per anchor, cooperative cancel, per-anchor exception isolation, defensive abort after 3 consecutive anchor failures (mark_banned; YandexRealtyScraper has no block/rate exception classes).
trigger_yandex_city_sweep_run + dispatch branch (scheduler.py): faithful mirror of the avito trigger.
Migration 078: seeds scrape_schedules row enabled=false, window 02:00-05:00 UTC, conservative {pages_per_anchor:2, request_delay_sec:9, radius_m:1500}, ON CONFLICT (source) DO NOTHING (idempotent). 2nd migration through the new _schema_migrations gate (#637).
8 unit tests — fully mocked (scraper + save_listings + scrape_runs), no live network/DB.

code-reviewer: ✅ approve, no blockers. Dormant-safety confirmed (get_due_schedules WHERE enabled=true + partial index → cannot fire); correctness-if-enabled traced; trigger session lifecycle clean; migration idempotent on unique source; zero shared-loop regression.

⚠️ Operator runbook (do NOT enable casually)

Live sweep hits Yandex from prod IP → anti-bot. Enable ONLY after #623 (egress-IP rotation):

UPDATE scrape_schedules SET enabled=true WHERE source='yandex_city_sweep';

Test plan

13 tests pass (8 new + 5 scheduler), ruff clean, py_compile clean
migration 078 seeds enabled=false (dormant)
Post-merge: deploy applies 078 once (tracked); scheduler stays healthy; NO yandex_city_sweep run fires (enabled=false)

Closes #561

## Summary Closes #561. Wires a full Yandex.Недвижимость periodic city-sweep into the in-app scheduler, **shipped DISABLED (enabled=false) by design** — live behavior is intentionally unverified pending egress-IP rotation (#623). Mirrors the existing `avito_city_sweep` (also currently disabled in prod). - **run_yandex_city_sweep** (scrape_pipeline.py): mirror of run_avito_city_sweep, simpler (no houses/detail enrich) — search + `save_listings` per anchor, cooperative cancel, per-anchor exception isolation, defensive abort after 3 consecutive anchor failures (`mark_banned`; YandexRealtyScraper has no block/rate exception classes). - **trigger_yandex_city_sweep_run** + dispatch branch (scheduler.py): faithful mirror of the avito trigger. - **Migration 078**: seeds scrape_schedules row `enabled=false`, window 02:00-05:00 UTC, conservative `{pages_per_anchor:2, request_delay_sec:9, radius_m:1500}`, `ON CONFLICT (source) DO NOTHING` (idempotent). 2nd migration through the new _schema_migrations gate (#637). - **8 unit tests** — fully mocked (scraper + save_listings + scrape_runs), no live network/DB. code-reviewer: ✅ approve, no blockers. Dormant-safety confirmed (`get_due_schedules WHERE enabled=true` + partial index → cannot fire); correctness-if-enabled traced; trigger session lifecycle clean; migration idempotent on unique `source`; zero shared-loop regression. ## ⚠️ Operator runbook (do NOT enable casually) Live sweep hits Yandex from prod IP → anti-bot. Enable ONLY after #623 (egress-IP rotation): ``` UPDATE scrape_schedules SET enabled=true WHERE source='yandex_city_sweep'; ``` ## Test plan - [x] 13 tests pass (8 new + 5 scheduler), ruff clean, py_compile clean - [x] migration 078 seeds enabled=false (dormant) - [ ] Post-merge: deploy applies 078 once (tracked); scheduler stays healthy; NO yandex_city_sweep run fires (enabled=false) Closes #561

Light1YT added 1 commit 2026-05-29 08:49:57 +00:00

feat(scheduler): add dormant Yandex Realty city-sweep (#561 ) ef0b0d0b3a

Wire a full Yandex.Недвижимость periodic city-sweep into the in-app
scheduler, shipped DISABLED (enabled=false) by design — live behavior is
intentionally unverified pending egress-IP rotation (#623). Mirrors the
existing avito_city_sweep (also currently disabled in prod).

- run_yandex_city_sweep in scrape_pipeline.py: mirrors run_avito_city_sweep
  but simpler (no houses/detail enrich) — search + save_listings per anchor,
  cooperative cancel, per-anchor exception isolation, defensive abort after
  3 consecutive anchor failures (mark_banned; YandexRealtyScraper has no
  block/rate exception classes — fetch_around swallows errors → []).
- trigger_yandex_city_sweep_run + dispatch branch in scheduler.py (faithful
  mirror of the avito trigger: claim run, fresh SessionLocal, create_task).
- Migration 078 seeds scrape_schedules row enabled=false, window 02:00-05:00
  UTC, conservative params {pages_per_anchor:2, request_delay_sec:9,
  radius_m:1500}, ON CONFLICT (source) DO NOTHING (idempotent).
- 8 unit tests (mocked scraper + save_listings, no live network/DB):
  anchor iteration, cancel mid-sweep, per-anchor isolation, consecutive-
  failure abort + reset-on-success, empty-page pagination break.

Enable later (after #623): UPDATE scrape_schedules SET enabled=true
WHERE source='yandex_city_sweep';

Closes #561