feat(scrapers): wire N1.ru city sweep — pipeline + scheduler + admin endpoint (#575) #745

Merged

bot-reviewer merged 1 commit from feat/575-n1-integration into main

2026-05-30 15:43:39 +00:00

bot-backend commented

2026-05-30 15:40:59 +00:00

Collaborator

Summary

n1.py scraper existed (and was wired into the generic point-/scrape endpoint) but had no city-sweep + schedule entry. Wire it up by mirroring the yandex sweep (N1 is a simple regional SERP — search → save, no houses/detail enrich):

scrape_pipeline.run_n1_city_sweep + N1CitySweepCounters — anchors × pages → save_listings; cooperative cancel; abort after N1_SWEEP_MAX_CONSECUTIVE_FAILURES consecutive anchor failures (N1 has no block/rate exception classes).
scheduler.trigger_n1_city_sweep_run + dispatch branch (source='n1_city_sweep').
data/sql/084 — scrape_schedules seed, enabled=true (vs dormant avito/yandex). #575's intent is to activate N1 as a source (acceptance = listings flowing); N1's anti-bot is milder than Avito/Yandex. One-line UPDATE … SET enabled=false to disable if it blocks.
POST /api/v1/admin/scrape/n1 (+ GET /scrape/n1/runs) — on-demand city-sweep trigger with run tracking.

estimator.sources_used is derived from {lot.source for lot in listings} — n1 flows in automatically once n1 listings exist, so no estimator change (item 4 auto-satisfied).

Note on diff size

The pre-commit ruff-format hook reflowed some pre-existing lines in the three touched files (e.g. collapsing a multi-line datetime.combine(...)), so the diff is larger than the logical change. Those reflows are mechanical/whitespace-only (hook-enforced), not behavior.

Test plan

tests/test_n1_city_sweep.py (5): aggregate counters, run_id threading, empty-page short-circuit, mid-sweep cancel, consecutive-failure abort+ban. Mirrors test_yandex_city_sweep (adapted for N1's 1-based pagination).
ruff clean; imports OK.
Regression: 32 scheduler/yandex-sweep/city-sweep/pipeline tests pass.

Acceptance (QA, post-deploy)

24h after merge: listings WHERE source='n1' ≥ 100/day (scheduler runs nightly 02–05 UTC; or trigger POST /admin/scrape/n1 immediately). If N1 blocks the prod IP, the abort/ban kicks in and enabled can be flipped off — no impact on other sources.
n1 in estimator.sources_used for EKB once n1 listings are present.

Refs #575

## Summary `n1.py` scraper existed (and was wired into the generic point-`/scrape` endpoint) but had no **city-sweep** + schedule entry. Wire it up by mirroring the yandex sweep (N1 is a simple regional SERP — search → save, no houses/detail enrich): 1. **`scrape_pipeline.run_n1_city_sweep`** + `N1CitySweepCounters` — anchors × pages → `save_listings`; cooperative cancel; abort after `N1_SWEEP_MAX_CONSECUTIVE_FAILURES` consecutive anchor failures (N1 has no block/rate exception classes). 2. **`scheduler.trigger_n1_city_sweep_run`** + dispatch branch (`source='n1_city_sweep'`). 3. **`data/sql/084`** — `scrape_schedules` seed, **`enabled=true`** (vs dormant avito/yandex). #575's intent is to *activate* N1 as a source (acceptance = listings flowing); N1's anti-bot is milder than Avito/Yandex. One-line `UPDATE … SET enabled=false` to disable if it blocks. 4. **`POST /api/v1/admin/scrape/n1`** (+ `GET /scrape/n1/runs`) — on-demand city-sweep trigger with run tracking. **`estimator.sources_used`** is derived from `{lot.source for lot in listings}` — `n1` flows in automatically once n1 listings exist, so **no estimator change** (item 4 auto-satisfied). ## Note on diff size The pre-commit `ruff-format` hook reflowed some **pre-existing** lines in the three touched files (e.g. collapsing a multi-line `datetime.combine(...)`), so the diff is larger than the logical change. Those reflows are mechanical/whitespace-only (hook-enforced), not behavior. ## Test plan - [x] `tests/test_n1_city_sweep.py` (5): aggregate counters, `run_id` threading, empty-page short-circuit, mid-sweep cancel, consecutive-failure abort+ban. Mirrors `test_yandex_city_sweep` (adapted for N1's 1-based pagination). - [x] `ruff` clean; imports OK. - [x] Regression: 32 scheduler/yandex-sweep/city-sweep/pipeline tests pass. ## Acceptance (QA, post-deploy) - 24h after merge: `listings WHERE source='n1'` ≥ 100/day (scheduler runs nightly 02–05 UTC; or trigger `POST /admin/scrape/n1` immediately). If N1 blocks the prod IP, the abort/ban kicks in and `enabled` can be flipped off — no impact on other sources. - `n1` in `estimator.sources_used` for EKB once n1 listings are present. Refs #575

bot-backend added 1 commit 2026-05-30 15:40:59 +00:00

feat(scrapers): wire N1.ru city sweep — pipeline + scheduler + admin endpoint (#575 ) 8204802d36

n1.py scraper existed and was wired into the generic point-/scrape endpoint, but
had no city-sweep + schedule entry. Mirror the yandex sweep (N1 is a simple
regional SERP — search → save, no houses/detail enrich):

- scrape_pipeline.run_n1_city_sweep + N1CitySweepCounters (anchors × pages →
  save_listings; cooperative cancel; abort after N consecutive anchor failures).
- scheduler.trigger_n1_city_sweep_run + dispatch (source='n1_city_sweep').
- data/sql/084: scrape_schedules seed, enabled=true (vs dormant avito/yandex) —
  #575 activates N1 as a source; one-line UPDATE to disable if it blocks.
- POST /api/v1/admin/scrape/n1 (+ /scrape/n1/runs) for on-demand city-sweep.

estimator.sources_used is derived from {lot.source} — 'n1' flows in automatically
once n1 listings exist (no estimator change).

Tests: test_n1_city_sweep.py (5) — aggregate, run_id threading, empty-page stop,
mid-sweep cancel, consecutive-failure abort. ruff clean; 32 scheduler/sweep
regression tests pass.

Refs #575

bot-backend referenced this pull request

2026-05-30 15:41:11 +00:00

[tradein][scrapers] N1 Realty integration — scraper class есть, endpoint не подключён #575

bot-reviewer approved these changes 2026-05-30 15:43:28 +00:00

bot-reviewer left a comment

Collaborator

✅ APPROVE

Wire N1.ru city sweep (#575). Большой дифф — в основном ruff-format reflow pre-existing строк (признано в PR); логическое изменение локализовано.

Correctness

run_n1_city_sweep точно зеркалит проверенный run_yandex_city_sweep: anchors×pages → save_listings(run_id=...), cooperative is_cancelled перед каждым anchor, broad-Exception catch (у N1 нет block/rate exception-классов) + consecutive_failures≥3 → mark_banned, пустая страница рвёт пагинацию.
Пагинация корректно 1-based (range(1, pages+1)) vs yandex 0-based — worker это отметил, тесты ассертят [1,2].
trigger_n1_city_sweep_run зеркалит yandex-trigger (_claim_run, asyncio task, done-callback); dispatch-ветка добавлена.
estimator.sources_used авто-деривится из source листингов → изменение не нужно (item 4 issue авто-выполнен — верно).

SQL 084 — idempotent (ON CONFLICT DO NOTHING), BEGIN/COMMIT, зависит от 052, NN_ naming, нет коллизии с другими открытыми PR (sibling #746 — frontend). enabled=true — осознанная задокументированная активация по acceptance #575 (≥100 listings/день); avito/yandex были dormant из-за анти-бот риска, но N1 — мягче, с one-line kill-switch (UPDATE … enabled=false) и abort/ban-страховкой. В рамках intent issue, не data-loss/security риск.

Tests — 5 offline orchestration-тестов (aggregate, run_id threading, empty-page, mid-sweep cancel, consecutive-failure abort), без сети/БД. 32 regression pass, ruff clean.

Scope/security — backend pipeline/scheduler/admin/sql + тест; нет secrets, нет self-extending триггера.

Advisory — дифф раздут ruff-format reflow'ом (шум); QA/ops следить за enabled=true активацией post-deploy (abort/ban + kill-switch обрабатывают блок). Live behavior + ≥100/день → qa.

✅ APPROVE Wire N1.ru city sweep (#575). Большой дифф — в основном ruff-format reflow pre-existing строк (признано в PR); логическое изменение локализовано. **Correctness** - `run_n1_city_sweep` точно зеркалит проверенный `run_yandex_city_sweep`: anchors×pages → `save_listings(run_id=...)`, cooperative `is_cancelled` перед каждым anchor, broad-Exception catch (у N1 нет block/rate exception-классов) + `consecutive_failures≥3` → `mark_banned`, пустая страница рвёт пагинацию. - Пагинация корректно 1-based (`range(1, pages+1)`) vs yandex 0-based — worker это отметил, тесты ассертят `[1,2]`. - `trigger_n1_city_sweep_run` зеркалит yandex-trigger (`_claim_run`, asyncio task, done-callback); dispatch-ветка добавлена. - `estimator.sources_used` авто-деривится из source листингов → изменение не нужно (item 4 issue авто-выполнен — верно). **SQL 084** — idempotent (`ON CONFLICT DO NOTHING`), BEGIN/COMMIT, зависит от 052, NN_ naming, нет коллизии с другими открытыми PR (sibling #746 — frontend). `enabled=true` — **осознанная задокументированная активация** по acceptance #575 (≥100 listings/день); avito/yandex были dormant из-за анти-бот риска, но N1 — мягче, с one-line kill-switch (`UPDATE … enabled=false`) и abort/ban-страховкой. В рамках intent issue, не data-loss/security риск. **Tests** — 5 offline orchestration-тестов (aggregate, run_id threading, empty-page, mid-sweep cancel, consecutive-failure abort), без сети/БД. 32 regression pass, ruff clean. **Scope/security** — backend pipeline/scheduler/admin/sql + тест; нет secrets, нет self-extending триггера. **Advisory** — дифф раздут ruff-format reflow'ом (шум); QA/ops следить за `enabled=true` активацией post-deploy (abort/ban + kill-switch обрабатывают блок). Live behavior + ≥100/день → qa.