|
|
ef0b0d0b3a
|
feat(scheduler): add dormant Yandex Realty city-sweep (#561)
Wire a full Yandex.Недвижимость periodic city-sweep into the in-app
scheduler, shipped DISABLED (enabled=false) by design — live behavior is
intentionally unverified pending egress-IP rotation (#623). Mirrors the
existing avito_city_sweep (also currently disabled in prod).
- run_yandex_city_sweep in scrape_pipeline.py: mirrors run_avito_city_sweep
but simpler (no houses/detail enrich) — search + save_listings per anchor,
cooperative cancel, per-anchor exception isolation, defensive abort after
3 consecutive anchor failures (mark_banned; YandexRealtyScraper has no
block/rate exception classes — fetch_around swallows errors → []).
- trigger_yandex_city_sweep_run + dispatch branch in scheduler.py (faithful
mirror of the avito trigger: claim run, fresh SessionLocal, create_task).
- Migration 078 seeds scrape_schedules row enabled=false, window 02:00-05:00
UTC, conservative params {pages_per_anchor:2, request_delay_sec:9,
radius_m:1500}, ON CONFLICT (source) DO NOTHING (idempotent).
- 8 unit tests (mocked scraper + save_listings, no live network/DB):
anchor iteration, cancel mid-sweep, per-anchor isolation, consecutive-
failure abort + reset-on-success, empty-page pagination break.
Enable later (after #623): UPDATE scrape_schedules SET enabled=true
WHERE source='yandex_city_sweep';
Closes #561
|
2026-05-29 13:49:28 +05:00 |
|
|
|
2914ff2ef2
|
feat(tradein): city sweep — auto ЕКБ pipeline (#477)
Deploy Trade-In / build-frontend (push) Successful in 23s
Deploy Trade-In / deploy (push) Successful in 22s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-backend (push) Successful in 38s
Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel).
- avito.py fetch_around(pages=N) — pagination backward compat default=1
- scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check
- scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent)
- admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel
- migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction)
- deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern)
- 16 offline tests
Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.
|
2026-05-23 14:38:07 +00:00 |
|
