feat(scraper-kit): copy sweep pipeline orchestration (avito) with protocol injection, strangler (#2135) #2171

Merged

lekss361 merged 1 commit from feat/scraper-kit-pipeline-copy into main

2026-07-02 17:08:41 +00:00

lekss361 commented

2026-07-02 17:08:33 +00:00

Owner

Часть #2135 (Epic #2126, шаг F). Strangler-copy — app/services/scrape_pipeline.py НЕ тронут, деплой инертен. Первый срез (avito) + инфра; остальные 7 sweep — follow-up.

Скопировано в `scraper_kit/orchestration/`

pipeline.py — run_avito_city_sweep (SEARCH→SAVE→GROUP→HOUSES→DETAIL + ban/rotation + partial-intake + IMV) + run_avito_pipeline + counters
runs.py — копия scrape_runs.py (sentry опционален)

Развязка + инфра

settings→config: ScraperConfig, matching→matcher: HouseMatcher, house_imv_backfill→enrichment: EnrichmentJobs
Расширил contracts.py: ScraperConfig +9 полей оркестрации + EnrichmentJobs.process_houses_imv_batch
Синхронизировал scraper_adapters.py (guard-тест зелёный, учтён урок #2170)
ban-rotation state-machine (3 инлайн-копии) → единый _try_rotate_within_budget

Golden-parity ✅ (6 сценариев)

Старый run_avito_city_sweep ≡ новый на мокнутом I/O: counters, full-ban→banned, partial-intake→done (флаг on/off), detail 3-block propagate, IMV counters.

Follow-up

7 остальных sweep (yandex/cian/domclick + 3 full_load + avito_newbuilding) — следующий срез.

Verify

боевые не тронуты; grep from app в orchestration пусто; ruff clean; 39 passed

Часть #2135 (Epic #2126, шаг F). **Strangler-copy** — `app/services/scrape_pipeline.py` НЕ тронут, деплой инертен. Первый срез (avito) + инфра; остальные 7 sweep — follow-up. ## Скопировано в `scraper_kit/orchestration/` - `pipeline.py` — `run_avito_city_sweep` (SEARCH→SAVE→GROUP→HOUSES→DETAIL + ban/rotation + partial-intake + IMV) + `run_avito_pipeline` + counters - `runs.py` — копия `scrape_runs.py` (sentry опционален) ## Развязка + инфра - `settings`→`config: ScraperConfig`, `matching`→`matcher: HouseMatcher`, `house_imv_backfill`→`enrichment: EnrichmentJobs` - Расширил `contracts.py`: `ScraperConfig` +9 полей оркестрации + `EnrichmentJobs.process_houses_imv_batch` - Синхронизировал `scraper_adapters.py` (guard-тест зелёный, учтён урок #2170) - ban-rotation state-machine (3 инлайн-копии) → единый `_try_rotate_within_budget` ## Golden-parity ✅ (6 сценариев) Старый `run_avito_city_sweep` ≡ новый на мокнутом I/O: counters, full-ban→banned, partial-intake→done (флаг on/off), detail 3-block propagate, IMV counters. ## Follow-up 7 остальных sweep (yandex/cian/domclick + 3 full_load + avito_newbuilding) — следующий срез. ## Verify - боевые не тронуты; grep `from app` в orchestration пусто; ruff clean; 39 passed

lekss361 added 1 commit 2026-07-02 17:08:33 +00:00