|
|
8204802d36
|
feat(scrapers): wire N1.ru city sweep — pipeline + scheduler + admin endpoint (#575)
n1.py scraper existed and was wired into the generic point-/scrape endpoint, but
had no city-sweep + schedule entry. Mirror the yandex sweep (N1 is a simple
regional SERP — search → save, no houses/detail enrich):
- scrape_pipeline.run_n1_city_sweep + N1CitySweepCounters (anchors × pages →
save_listings; cooperative cancel; abort after N consecutive anchor failures).
- scheduler.trigger_n1_city_sweep_run + dispatch (source='n1_city_sweep').
- data/sql/084: scrape_schedules seed, enabled=true (vs dormant avito/yandex) —
#575 activates N1 as a source; one-line UPDATE to disable if it blocks.
- POST /api/v1/admin/scrape/n1 (+ /scrape/n1/runs) for on-demand city-sweep.
estimator.sources_used is derived from {lot.source} — 'n1' flows in automatically
once n1 listings exist (no estimator change).
Tests: test_n1_city_sweep.py (5) — aggregate, run_id threading, empty-page stop,
mid-sweep cancel, consecutive-failure abort. ruff clean; 32 scheduler/sweep
regression tests pass.
Refs #575
|
2026-05-30 18:39:36 +03:00 |
|
